Back to search

Cornell University

Scaling the Efficiency of Large-Scale Learning with Structured Representations

Abstract

dc:description.abstract

One overarching theme of the past decade of machine learning is that scaling any of the many axes of modeling generally improves model quality. By increasing data, compute, or the number of parameters, we can usually expect improvements in downstream performance. To the casual observer, it may seem that large, unstructured models are the key to deep learning. However, careful analysis shows that modern models are actually rich in structure. In this thesis, I show that it is possible to exploit these structures to improve the efficiency of deep models. In Part 2, I demonstrate that by using the geometric properties of Hyperbolic space to model hierarchies in attention mechanisms, we can significantly improve modeling performance on hierarchical data. In Part 3, I consider the problem of LLM quantization. I first show that by taking advantage of both the geometric structure of the model weights as well as the model loss landscape, we can compress models to as little as 1/8 of their original size without catastrophic loss of fidelity. Then, I apply these concepts to low-precision training, where I introduce the first near-lossless FP4 LLM training recipe, accelerating training by over 2x. Finally, in Part 4, I explore explicitly embedding structure into models with sparse embedding tables, resulting in more performant and faster structured sparse models. Many of these findings have been adopted in industry, confirming that even in the age of scaling, structural information is crucial for efficient learning.

Degree

thesis:*
Name thesis:degree_name
Ph. D., Computer Science
Level thesis:degree_level
Doctor of Philosophy
Discipline thesis:degree_discipline
Computer Science
Grantor
Cornell University
Year dc:date.issued
2026

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Tseng, Albert
Chair dc:contributor.chair
  • De Sa, Christopher
Committee members dc:contributor.committeemember
  • Bindel, David
  • Ellis, Kevin

Subjects

dc:subject × 3

Rights

Language dc:language.iso
en

Identifiers

dc:identifier.*
Dc Identifier Other
Tseng_cornellgrad_0058F_15467
https://www.proquest.com/LegacyDocView/DISSNUM/32574126

Chain of custody

source
Harvested from
Cornell University
Base URL
ecommons.cornell.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Tseng, Albert. Scaling the Efficiency of Large-Scale Learning with Structured Representations. Doctor of Philosophy thesis, Cornell University, 2026. https://hdl.handle.net/1813/126519