Back to search

Cornell University

Foundations of Diffusion Language Models

Abstract

dc:description.abstract

Diffusion models have recently emerged as a powerful alternative to autoregressive (AR) models for generative modeling, with strong results in continuous domains such as images and video. In the discrete setting of language, however, diffusion models still lag behind AR approaches in both likelihood and sampling efficiency. This thesis investigates the foundations of diffusion language models and asks whether they can be made competitive with traditional AR language models. First, we revisit the role of the forward noising process. Contrary to the prevailing theory that likelihood is invariant to the noise schedule, we prove that this invariance holds only for univariate schedules. We introduce context-adaptive, multivariate noise processes that are learned from data and show that they strictly improve likelihood and sample quality. Second, we propose a simple and unified framework for discrete diffusion that encompasses masked and Uniform-state processes. Within this framework, we derive Rao–Blackwellized variational bounds that are tighter and exhibit lower variance than existing ELBO formulations. Using these bounds, our MDLM models achieve state-of-the-art diffusion perplexities, approach AR perplexities on standard language benchmarks, and surpass AR models on several likelihood evaluations, while enabling efficient conversion of BERT-style encoders into generative models. Third, we establish a discrete–continuous duality that links Uniform-state diffusion in discrete space with Gaussian diffusion in continuous space. This connection allows us to transfer efficient parameterizations, improved training objectives, and distillation-based fast samplers to the discrete domain. Leveraging this, we obtain a twofold speedup in training convergence and two orders of magnitude faster sampling, and we introduce the first few-step discrete diffusion samplers via discrete consistency distillation. Collectively, these contributions show that diffusion language models can substantially close the gap to AR models in both quality and speed, while providing a principled foundation for future advances in discrete diffusion, fast sampling, and controllable language generation.

Degree

thesis:*
Name thesis:degree_name
Ph. D., Computer Science
Level thesis:degree_level
Doctor of Philosophy
Discipline thesis:degree_discipline
Computer Science
Grantor
Cornell University
Year dc:date.issued
2026

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Sahoo, Subham
Chair dc:contributor.chair
  • Thickstun, John
Committee members dc:contributor.committeemember
  • Snavely, Keith
  • Selman, Bart

Subjects

dc:subject × 3

Rights

dc:rights
Statement dc:rights
  • Attribution 4.0 International
Language dc:language.iso
en

Identifiers

dc:identifier.*
Dc Identifier Other
Sahoo_cornellgrad_0058F_15451
https://www.proquest.com/LegacyDocView/DISSNUM/32447506

Chain of custody

source
Harvested from
Cornell University
Base URL
ecommons.cornell.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Sahoo, Subham. Foundations of Diffusion Language Models. Doctor of Philosophy thesis, Cornell University, 2026. https://hdl.handle.net/1813/126638