Back to search

University of British Columbia

The implicit geometry of language : structure, semantics, and dynamics in next-token prediction

Abstract

dc:description

Language models trained with next-token prediction acquire internal representations that encode rich semantic structure, including syntactic regularities, linear semantic analogies, and a coarse-to-fine progression in learned concepts. While these phenomena are well documented, their origin remains conceptually opaque: how does predicting next-token distributions produce such structure? This thesis tackles this question from first principles. We frame next-token prediction as a soft-label classification problem, where each context is paired with a sparse target distribution over next tokens. Variation in support across contexts creates a structured pattern of label sparsity, captured by a data co-occurrence matrix. To formalize this, we adopt the Unconstrained Features Model (UFM), a minimal setting where logits are parameterized as a product of word and context embeddings and trained with cross-entropy loss plus ridge regularization. We prove that gradient descent drives directional convergence: context embeddings with identical support sets collapse to a shared direction, partitioning the embedding space by support pattern. This process uncovers latent structure in the data, even without explicit semantic supervision. We further show that the singular vectors of the co-occurrence matrix align with interpretable semantic directions in embedding space. Each vector induces a partition over words or contexts based on alignment with a latent concept. These directions emerge sequentially during training, with dominant concepts learned first—yielding a spectral explanation for the progression from coarse to fine-grained structure. Experiments on synthetic and real data confirm these predictions. Trained transformers exhibit the same alignment between embeddings and the singular directions of the co-occurrence matrix. Semantic structure appears in both synthetic tasks and pretrained models. Tracking learning in text data and step-imbalanced MNIST shows coarse distinctions—such as part-of-speech or majority class—emerging early, followed by finer categories, consistent with spectral ordering. Finally, we extend the analysis to settings with class imbalance, showing that reweighting the loss flattens the singular spectrum of the co-occurrence matrix, thereby accelerating the acquisition of underrepresented features. These results demonstrate that semantic structure in next-token prediction arises predictably from the interaction between label structure—particularly its spectral properties—and the implicit bias of optimization.

Degree

thesis:*
Name thesis:degree_name
Master of Applied Science - MASc
Level thesis:degree_level
master's
Discipline thesis:degree_discipline
Electrical and Computer Engineering
Grantor dc:publisher
University of British Columbia
Year dc:date
2025

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Zhao, Yize
Contributors dc:contributor
  • Thrampoulidis, Christos

Rights

dc:rights
Statement dc:rights
  • Attribution-NonCommercial-NoDerivatives 4.0 International
Language dc:language
eng

Identifiers

dc:identifier.*
Handle dc:identifier
http://hdl.handle.net/2429/92005

Chain of custody

source
Harvested from
University of British Columbia
Base URL
circle.library.ubc.ca/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
related terms
citation

Zhao, Yize. The implicit geometry of language : structure, semantics, and dynamics in next-token prediction. master's thesis, University of British Columbia, 2025. http://hdl.handle.net/2429/92005