Cornell University
Scaling the Efficiency of Large-Scale Learning with Structured Representations
Abstract
dc:description.abstractOne overarching theme of the past decade of machine learning is that scaling any of the many axes of modeling generally improves model quality. By increasing data, compute, or the number of parameters, we can usually expect improvements in downstream performance. To the casual observer, it may seem that large, unstructured models are the key to deep learning. However, careful analysis shows that modern models are actually rich in structure. In this thesis, I show that it is possible to exploit these structures to improve the efficiency of deep models. In Part 2, I demonstrate that by using the geometric properties of Hyperbolic space to model hierarchies in attention mechanisms, we can significantly improve modeling performance on hierarchical data. In Part 3, I consider the problem of LLM quantization. I first show that by taking advantage of both the geometric structure of the model weights as well as the model loss landscape, we can compress models to as little as 1/8 of their original size without catastrophic loss of fidelity. Then, I apply these concepts to low-precision training, where I introduce the first near-lossless FP4 LLM training recipe, accelerating training by over 2x. Finally, in Part 4, I explore explicitly embedding structure into models with sparse embedding tables, resulting in more performant and faster structured sparse models. Many of these findings have been adopted in industry, confirming that even in the age of scaling, structural information is crucial for efficient learning.
Degree
thesis:*- Name thesis:degree_name
- Ph. D., Computer Science
- Level thesis:degree_level
- Doctor of Philosophy
- Discipline thesis:degree_discipline
- Computer Science
- Grantor
- Cornell University
- Year dc:date.issued
- 2026
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Tseng, Albert
- Chair dc:contributor.chair
-
- De Sa, Christopher
- Committee members dc:contributor.committeemember
-
- Bindel, David
- Ellis, Kevin
Subjects
dc:subject × 3Rights
- Language dc:language.iso
- en
Identifiers
dc:identifier.*- Dc Identifier Other
-
Tseng_cornellgrad_0058F_15467
https://www.proquest.com/LegacyDocView/DISSNUM/32574126