Back to search

Cornell University

Implicit Regularization and Inductive Biases in Neural Network Training

Abstract

dc:description.abstract

Artificial intelligence has advanced rapidly in recent years, driven largely by the success of neural networks trained at unprecedented scale.A defining feature of these models is their scalability: as model size, data, and compute increase, performance often improves in a predictable manner, enabling the solution of increasingly complex problems. Despite this success, fundamental questions remain. Neural networks are typically highly overparameterized and trained on noisy, finite datasets, yet they often generalize well without explicit regularization. Understanding why this is possible remains a central challenge in modern machine learning. Two closely related concepts play a key role in addressing this challenge: implicit regularization and inductive bias. Implicit regularization refers to the tendency of the training process itself --- through optimization algorithms, parameterization, and initialization --- to favor certain solutions over others, even in the absence of explicit regularization terms. Inductive biases describe the structural preferences encoded by a model’s architecture and design choices, which guide learning toward solutions that generalize beyond the training data. This dissertation focuses on understanding these two aspects of neural network training.We study the implicit regularization and inductive biases of several widely used model classes, including multi-layer perceptrons, state-space models, and Transformer architectures. Across these settings, we analyze how frequency structure, rank structure, and long-range dependencies emerge during training, and how they are shaped by architectural choices and parameterization. Importantly, we go beyond characterization and show how these effects can be tuned in a principled way through data design, model construction, and training procedures. By clarifying how implicit regularization and inductive biases arise and evolve, this work provides a unified perspective on the generalization behavior of modern neural networks and offers practical guidance for designing scalable models that are better aligned with the structure of the tasks they are intended to solve.

Degree

thesis:*
Name thesis:degree_name
Ph. D., Applied Mathematics
Level thesis:degree_level
Doctor of Philosophy
Discipline thesis:degree_discipline
Applied Mathematics
Grantor
Cornell University
Year dc:date.issued
2026

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Yu, Annan
Chair dc:contributor.chair
  • Townsend, Alex
Committee members dc:contributor.committeemember
  • Damle, Anil
  • Dean, Sarah

Subjects

dc:subject × 6

Rights

dc:rights
Statement dc:rights
  • Attribution 4.0 International
Language dc:language.iso
en

Identifiers

dc:identifier.*
Dc Identifier Other
Yu_cornellgrad_0058F_15458
https://www.proquest.com/LegacyDocView/DISSNUM/32449858

Chain of custody

source
Harvested from
Cornell University
Base URL
ecommons.cornell.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Yu, Annan. Implicit Regularization and Inductive Biases in Neural Network Training. Doctor of Philosophy thesis, Cornell University, 2026. https://hdl.handle.net/1813/126569