Cornell University
Implicit Regularization and Inductive Biases in Neural Network Training
Abstract
dc:description.abstractArtificial intelligence has advanced rapidly in recent years, driven largely by the success of neural networks trained at unprecedented scale.A defining feature of these models is their scalability: as model size, data, and compute increase, performance often improves in a predictable manner, enabling the solution of increasingly complex problems. Despite this success, fundamental questions remain. Neural networks are typically highly overparameterized and trained on noisy, finite datasets, yet they often generalize well without explicit regularization. Understanding why this is possible remains a central challenge in modern machine learning. Two closely related concepts play a key role in addressing this challenge: implicit regularization and inductive bias. Implicit regularization refers to the tendency of the training process itself --- through optimization algorithms, parameterization, and initialization --- to favor certain solutions over others, even in the absence of explicit regularization terms. Inductive biases describe the structural preferences encoded by a model’s architecture and design choices, which guide learning toward solutions that generalize beyond the training data. This dissertation focuses on understanding these two aspects of neural network training.We study the implicit regularization and inductive biases of several widely used model classes, including multi-layer perceptrons, state-space models, and Transformer architectures. Across these settings, we analyze how frequency structure, rank structure, and long-range dependencies emerge during training, and how they are shaped by architectural choices and parameterization. Importantly, we go beyond characterization and show how these effects can be tuned in a principled way through data design, model construction, and training procedures. By clarifying how implicit regularization and inductive biases arise and evolve, this work provides a unified perspective on the generalization behavior of modern neural networks and offers practical guidance for designing scalable models that are better aligned with the structure of the tasks they are intended to solve.
Degree
thesis:*- Name thesis:degree_name
- Ph. D., Applied Mathematics
- Level thesis:degree_level
- Doctor of Philosophy
- Discipline thesis:degree_discipline
- Applied Mathematics
- Grantor
- Cornell University
- Year dc:date.issued
- 2026
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Yu, Annan
- Chair dc:contributor.chair
-
- Townsend, Alex
- Committee members dc:contributor.committeemember
-
- Damle, Anil
- Dean, Sarah
Subjects
dc:subject × 6Rights
dc:rights- Statement dc:rights
-
- Attribution 4.0 International
- Licence dc:rights.uri
- Language dc:language.iso
- en
Identifiers
dc:identifier.*- Dc Identifier Other
-
Yu_cornellgrad_0058F_15458
https://www.proquest.com/LegacyDocView/DISSNUM/32449858