Back to search

Georgia Institute of Technology

Learning Implicitly Biased Overparameterized Models

Abstract

dc:description.abstract

The success of modern machine learning is built upon large-scale, highly parameterized models. In the overparameterized regime, these models routinely achieve near-zero training loss, yet generalize remarkably well to unseen data. A central mechanism underlying this phenomenon is implicit bias: gradient-based optimization does not converge to an arbitrary minimizer, but rather systematically favors solutions with particular structures or properties, determined by the interplay of the loss function, model architecture, and data distribution. This thesis studies the implicit bias of gradient descent (GD) in the overparameterized regime from two complementary perspectives: characterizing the mathematical and geometric structure of the solutions selected by the optimizer, and understanding the downstream consequences of this structure for generalization across diverse scenarios, including in-distribution (ID) and out-of-distribution (OOD) settings. In the first part, we characterize the emergence and geometry of implicit bias across a progression of settings with increasing complexity. For linear regression, gradient descent initialized at the origin converges to the minimum-l2-norm interpolation (MNI) solution for the entire class of unique finite-root loss functions. For linear classification, we extend the classical max-margin result for exponentially-tailed losses to the broader class of strictly monotone losses, including polynomially-tailed losses. By presenting matching upper and lower bounds, we show that in sufficiently high dimensions, the implicit bias of any such loss is approximately equivalent to the MNI. We then extend these results to nonlinear models: for shallow ReLU networks in the overparameterized regime, we demonstrate that gradient descent from small positive initializations converges to solutions that approximate MNI in regression tasks. Furthermore, to complete our characterization of the nonlinear regime, we summarize established foundational results for homogeneous networks under classification losses, reviewing their convergence to KKT points and the resulting margin structure. In the second part, we investigate the generalization consequences of these implicitly biased solutions. We first survey theoretical frameworks for generalization analysis by detailing two distinct analytical approaches: the exact characterization of closed-form solutions and the continuous analysis of gradient dynamics. We then formalize the distinction between ID and OOD generalization, analyzing how implicit bias interacts with covariate, label, or concept shifts. For OOD classification under spurious correlations, we analyze the disparate impact on majority and minority groups, evaluate importance-weighting as a corrective mechanism, and derive group-wise generalization bounds. Finally, we introduce and analyze the novel problem of task shift: transferring a representation learned for a classification task directly to a regression task. We rigorously characterize the conditions under which zero-shot and few-shot task shifts succeed, formally connecting the MNI implicit bias of classification to downstream regression performance. Taken together, the results in this thesis reveal a unifying picture: gradient descent in overparameterized models inherently tends toward minimum-norm solutions across a wide range of settings. Understanding this bias precisely is key to explaining both the benign generalization behavior observed in-distribution and the relative fragility or robustness of models under distribution shift.

Degree

thesis:*
Name thesis:degree_name
Electrical and Computer Engineering, PhD
Department thesis:degree_department
Electrical and Computer Engineering
Grantor
Georgia Institute of Technology
Year dc:date.issued
2026

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Lai, Kuo-Wei
Advisor dc:contributor.advisor
  • Muthukumar, Vidya
Committee members dc:contributor.committeemember
  • Tao, Molei
  • Davenport, Mark
  • Romberg, Justin

Identifiers

dc:identifier.*
Handle dc:identifier.uri
https://hdl.handle.net/1853/82187

Chain of custody

source
Harvested from
Georgia Tech
Base URL
repository.gatech.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
related terms
citation

Lai, Kuo-Wei. Learning Implicitly Biased Overparameterized Models. Georgia Institute of Technology, 2026. https://hdl.handle.net/1853/82187