University of Minnesota
Adaptive Iterative Methods for Nonlinear Equations, Nonlinear Eigenvalue Problems, and Deep Learning
Abstract
dc:description.abstractIterative methods built on matrix computations are standard tools for solving large-scale problems across science, engineering, and machine learning. Many of their foundational techniques were developed for linear systems and linear eigenvalue problems, while much of modern practice is nonlinear: equations that model physical equilibria are typically nonlinear, training a large language model minimizes a nonconvex objective, and many eigenvalue problems arising in engineering depend nonlinearly on the eigenvalue. This dissertation develops iterative methods for all three settings: acceleration methods for nonlinear equations and optimization, matrix-structured optimizers for large language model training, and projection methods for nonlinear eigenvalue problems. Their relationship to numerical linear algebra takes different forms: some extend linear iterative methods directly, while others modify matrix-based operators to meet the requirements of nonlinear computation. They nevertheless share a design pattern: preserve selected properties of an existing method, introduce a computational simplification, and permit an important algorithmic choice to be revised during execution. In the first part, the dissertation presents two acceleration methods for nonlinear equations. nlTGCR extends the truncated generalized conjugate residual method from linear systems to nonlinear equations, with an adaptive rule that switches to a cheaper linearized mode when the frozen local model remains accurate, reducing the function-evaluation cost over each linearized phase. A direction check uses projected curvature to revise updates that are unsuitable for descent on nonconvex objectives. AATGS brings the same orthogonalization technique to Anderson Acceleration, reformulating it on an orthogonal truncated Gram-Schmidt basis; it admits a short-term recurrence for linear symmetric problems and enhances numerical stability through an automatic restart mechanism triggered by an inexpensive error monitor. The second part follows this line from nonlinear optimization into deep learning. The dissertation presents HiMuon, a variant of the Muon optimizer for large language model training. HiMuon introduces a tile size into Muon's Newton-Schulz update. Its full-matrix setting recovers Muon exactly, while smaller tiles trade interactions across tile boundaries for lower arithmetic cost. Cross-parameter batching and tile-size-specific kernels improve the execution of these smaller matrix operations, and runtime reconfiguration allows the trade-off to change across training phases. Experiments show training losses close to those of Muon together with lower Newton-Schulz and end-to-end step times during the tiled phases. In the third part, the dissertation presents a projection method for nonlinear eigenvalue problems based on a combined rational-Chebyshev approximation and linearization. The method places shifts automatically and refines them during the iteration, reducing the manual spectral information required by shift-and-invert and rational-Krylov projection methods. A reduced subspace iteration works with short vector components of the linearization, and the residual analysis separates the effects of approximation and projection. Experiments on three benchmark problems demonstrate automatic shift refinement and small residuals for the targeted eigenpairs. Together, the four methods show how numerical linear algebra can support nonlinear computation through both direct extensions of linear methods and modifications of matrix-based operators for new mathematical and computational requirements. Across these different relationships, preserving the appropriate properties, controlling computational cost, and revising selected choices at runtime form a common basis for practical algorithm design.
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Tang, Ziyuan
Rights
- Language dc:language.iso
- en
Identifiers
dc:identifier.*- Dc Identifier Citation
- Tang, Ziyuan. (2026). Adaptive Iterative Methods for Nonlinear Equations, Nonlinear Eigenvalue Problems, and Deep Learning. Retrieved from the University Digital Conservancy, https://hdl.handle.net/11299/283156.