Abstract
dc:description.abstractThe rapid scaling of deep neural networks has led to unprecedented success across a wide range of application domains. However, the growing size and complexity of modern foundation models introduce significant challenges in terms of training scalability, stability, and efficiency, imposing severe computational and memory bottlenecks on both pre-training and fine-tuning. The prohibitive resources demanded by foundation models make naive scaling infeasible, and motivate the design of principled training strategies that leverage model structure for scalability and efficiency. This thesis addresses these challenges by jointly designing network architectures and tailoring optimization to accelerate convergence and enhance training efficiency. Chapter 1 offers an introduction to modern foundation models, the training pipeline, and underlying challenges. On the one hand, scalability (especially scaling depth) is the key hurdle during pre-training, typically mitigated via residual connections. However, strong evidence suggests that conventional residual connections can lead to underutilization of neural network depth. On the other hand, the primary challenge in fine-tuning is computational and memory efficiency. While low-rank adaptation (LoRA) can be utilized to alleviate the computational overhead, it suffers from slow optimization convergence and notable performance drop. Chapter 2 analyzes an often overlooked perspective, namely residual topology, in model scaling. A case study illustrates that different residual topologies can induce exponentially different convergence rates. Building upon this insight, we investigate adaptive neural connection reassignment (ANCRe), a data-driven residual architecture that parameterizes and learns an optimal residual layout on the fly. ANCRe utilizes properly normalized coefficients to adaptively reweight residual shortcuts with marginal computational overhead, yet significantly accelerates the pre-training of foundation models. This results in effective utilization of network depth, thereby achieving the highly desired scalability. Chapter 3 focuses on the optimization of LoRA. The non-uniqueness of LoRA’s low-rank factorization introduces a gauge symmetry that decelerates its convergence. To address this issue, the thesis develops refactored low-rank adaptation (RefLoRA), a symmetry-aware optimization scheme that identifies the optimal low-rank decomposition minimizing the loss. This yields gauge-invariant optimization without "gauge shift," that is, ineffective moves that do not alter the loss, thus speeding up convergence and stabilizing LoRA fine-tuning. Chapter 4 deals with the efficiency-expressiveness tradeoff induced by LoRA's low-rank premise. A sufficient and necessary condition for the optimal low-rank subspace update is derived, which demands truncated singular value decomposition (SVD) with prohibitive computational overhead. To circumvent this, the thesis shows that the optimal subspace can be readily acquired by constraining the low-rank adapters. This design efficiently accumulates an effectively high-rank weight update without truncated SVD and resetting optimizer state. Taken together, this thesis embodies depth-aware architectural design alongside symmetry- and rank-aware optimization. The proposed methods yield consistent gains in convergence speed and task performance across various setups. Ultimately, the research outcome of this thesis demonstrates that the joint design of residual architectures and low-rank optimization provides a substantially more scalable and efficient framework for training foundation models.
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Zhang, Yilang
Subjects
dc:subject × 6Rights
- Language dc:language.iso
- en
Identifiers
dc:identifier.*- Dc Identifier Citation
- Zhang, Yilang. (2026). Scalable and Efficient Training of Foundation Models. Retrieved from the University Digital Conservancy, https://hdl.handle.net/11299/280361.