Abstract
dc:description.abstractThe success of modern machine learning is built upon large-scale, highly parameterized models. In the overparameterized regime, these models routinely achieve near-zero training loss, yet generalize remarkably well to unseen data. A central mechanism underlying this phenomenon is implicit bias: gradient-based optimization does not converge to an arbitrary minimizer, but rather systematically favors solutions with particular structures or properties, determined by the interplay of the loss function, model architecture, and data distribution. This thesis studies the implicit bias of gradient descent (GD) in the overparameterized regime from two complementary perspectives: characterizing the mathematical and geometric structure of the solutions selected by the optimizer, and understanding the downstream consequences of this structure for generalization across diverse scenarios, including in-distribution (ID) and out-of-distribution (OOD) settings. In the first part, we characterize the emergence and geometry of implicit bias across a progression of settings with increasing complexity. For linear regression, gradient descent initialized at the origin converges to the minimum-l2-norm interpolation (MNI) solution for the entire class of unique finite-root loss functions. For linear classification, we extend the classical max-margin result for exponentially-tailed losses to the broader class of strictly monotone losses, including polynomially-tailed losses. By presenting matching upper and lower bounds, we show that in sufficiently high dimensions, the implicit bias of any such loss is approximately equivalent to the MNI. We then extend these results to nonlinear models: for shallow ReLU networks in the overparameterized regime, we demonstrate that gradient descent from small positive initializations converges to solutions that approximate MNI in regression tasks. Furthermore, to complete our characterization of the nonlinear regime, we summarize established foundational results for homogeneous networks under classification losses, reviewing their convergence to KKT points and the resulting margin structure. In the second part, we investigate the generalization consequences of these implicitly biased solutions. We first survey theoretical frameworks for generalization analysis by detailing two distinct analytical approaches: the exact characterization of closed-form solutions and the continuous analysis of gradient dynamics. We then formalize the distinction between ID and OOD generalization, analyzing how implicit bias interacts with covariate, label, or concept shifts. For OOD classification under spurious correlations, we analyze the disparate impact on majority and minority groups, evaluate importance-weighting as a corrective mechanism, and derive group-wise generalization bounds. Finally, we introduce and analyze the novel problem of task shift: transferring a representation learned for a classification task directly to a regression task. We rigorously characterize the conditions under which zero-shot and few-shot task shifts succeed, formally connecting the MNI implicit bias of classification to downstream regression performance. Taken together, the results in this thesis reveal a unifying picture: gradient descent in overparameterized models inherently tends toward minimum-norm solutions across a wide range of settings. Understanding this bias precisely is key to explaining both the benign generalization behavior observed in-distribution and the relative fragility or robustness of models under distribution shift.
Degree
thesis:*- Name thesis:degree_name
- Electrical and Computer Engineering, PhD
- Department thesis:degree_department
- Electrical and Computer Engineering
- Grantor
- Georgia Institute of Technology
- Year dc:date.issued
- 2026
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Lai, Kuo-Wei
- Advisor dc:contributor.advisor
-
- Muthukumar, Vidya
- Committee members dc:contributor.committeemember
-
- Tao, Molei
- Davenport, Mark
- Romberg, Justin
Identifiers
dc:identifier.*- Handle dc:identifier.uri
- https://hdl.handle.net/1853/82187