Back to search

University of Minnesota

Adaptive Iterative Methods for Nonlinear Equations, Nonlinear Eigenvalue Problems, and Deep Learning

Abstract

dc:description.abstract

Iterative methods built on matrix computations are standard tools for solving large-scale problems across science, engineering, and machine learning. Many of their foundational techniques were developed for linear systems and linear eigenvalue problems, while much of modern practice is nonlinear: equations that model physical equilibria are typically nonlinear, training a large language model minimizes a nonconvex objective, and many eigenvalue problems arising in engineering depend nonlinearly on the eigenvalue. This dissertation develops iterative methods for all three settings: acceleration methods for nonlinear equations and optimization, matrix-structured optimizers for large language model training, and projection methods for nonlinear eigenvalue problems. Their relationship to numerical linear algebra takes different forms: some extend linear iterative methods directly, while others modify matrix-based operators to meet the requirements of nonlinear computation. They nevertheless share a design pattern: preserve selected properties of an existing method, introduce a computational simplification, and permit an important algorithmic choice to be revised during execution. In the first part, the dissertation presents two acceleration methods for nonlinear equations. nlTGCR extends the truncated generalized conjugate residual method from linear systems to nonlinear equations, with an adaptive rule that switches to a cheaper linearized mode when the frozen local model remains accurate, reducing the function-evaluation cost over each linearized phase. A direction check uses projected curvature to revise updates that are unsuitable for descent on nonconvex objectives. AATGS brings the same orthogonalization technique to Anderson Acceleration, reformulating it on an orthogonal truncated Gram-Schmidt basis; it admits a short-term recurrence for linear symmetric problems and enhances numerical stability through an automatic restart mechanism triggered by an inexpensive error monitor. The second part follows this line from nonlinear optimization into deep learning. The dissertation presents HiMuon, a variant of the Muon optimizer for large language model training. HiMuon introduces a tile size into Muon's Newton-Schulz update. Its full-matrix setting recovers Muon exactly, while smaller tiles trade interactions across tile boundaries for lower arithmetic cost. Cross-parameter batching and tile-size-specific kernels improve the execution of these smaller matrix operations, and runtime reconfiguration allows the trade-off to change across training phases. Experiments show training losses close to those of Muon together with lower Newton-Schulz and end-to-end step times during the tiled phases. In the third part, the dissertation presents a projection method for nonlinear eigenvalue problems based on a combined rational-Chebyshev approximation and linearization. The method places shifts automatically and refines them during the iteration, reducing the manual spectral information required by shift-and-invert and rational-Krylov projection methods. A reduced subspace iteration works with short vector components of the linearization, and the residual analysis separates the effects of approximation and projection. Experiments on three benchmark problems demonstrate automatic shift refinement and small residuals for the targeted eigenpairs. Together, the four methods show how numerical linear algebra can support nonlinear computation through both direct extensions of linear methods and modifications of matrix-based operators for new mathematical and computational requirements. Across these different relationships, preserving the appropriate properties, controlling computational cost, and revising selected choices at runtime form a common basis for practical algorithm design.

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Tang, Ziyuan

Rights

Language dc:language.iso
en

Identifiers

dc:identifier.*
Dc Identifier Citation
Tang, Ziyuan. (2026). Adaptive Iterative Methods for Nonlinear Equations, Nonlinear Eigenvalue Problems, and Deep Learning. Retrieved from the University Digital Conservancy, https://hdl.handle.net/11299/283156.

Chain of custody

source
Harvested from
University of Minnesota
Base URL
conservancy.umn.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
related terms
citation

Tang, Ziyuan. Adaptive Iterative Methods for Nonlinear Equations, Nonlinear Eigenvalue Problems, and Deep Learning. 2026. https://hdl.handle.net/11299/283156