Back to search

Cornell University

Linear recurrent models for robust and interpretable long-context modeling

Abstract

dc:description.abstract

Transformers, powered by self-attention, have become foundational across modalities due to their scalability through parallelism. However, many real-world tasks—from byte-level language modeling to retrieval and reasoning—demand processing extended contexts, often spanning tens of thousands of tokens. The quadratic complexity of Transformers during training and linear KV-cache growth at inference makes them computationally prohibitive for such tasks, necessitating the exploration of more efficient architectures. Linear-time recurrent models and hybrid architectures, which combine linear recurrence with local attention, have emerged as promising alternatives to address these challenges. This thesis investigates two interconnected dimensions of long-context modeling: the development of scalable architectures and the interpretability of their underlying recurrence mechanisms. In the context of scalable architectures, we adapt the Mamba linear recurrent byte-level sequence modeling, tackling the computational challenges posed by extended sequence lengths. By introducing speculative decoding that combines subword drafting with byte-level verification, we demonstrate that token-free models can achieve subword-like inference speedups while maintaining robustness to noise and generalization over long sequences. On the interpretability front, we focus on hybrid models like RecurrentGemma, where recurrent hidden states serve as memory to bridge local attention windows. Using sparse autoencoders, we extract interpretable features from these recurrent states to explore their role in retaining long-range information. Through tasks such as long-context passkey retrieval, we reveal how these mechanisms facilitate extended context dependencies and offer insights into the memory of these fixed-memory models. Through these contributions, this thesis advances the understanding and development of architectures that combine scalability and interpretability for long-context modeling.

Degree

thesis:*
Name thesis:degree_name
M.S., Computer Science
Level thesis:degree_level
Master of Science
Discipline thesis:degree_discipline
Computer Science
Grantor
Cornell University
Year dc:date.issued
2024

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Gangavarapu, Tushaar
Chair dc:contributor.chair
  • Rush, Alexander
Committee member dc:contributor.committeemember
  • Bindel, David

Subjects

dc:subject × 5

Rights

dc:rights
Statement dc:rights
  • Attribution 4.0 International
Language dc:language.iso
en

Identifiers

dc:identifier.*
Dc Identifier Bibid
16921940
Dc Identifier Other
ProQuest Submission ID: 12319
ProQuest Publication ID: 31641441

Chain of custody

source
Harvested from
Cornell University
Base URL
ecommons.cornell.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Gangavarapu, Tushaar. Linear recurrent models for robust and interpretable long-context modeling. Master of Science thesis, Cornell University, 2024. https://hdl.handle.net/1813/117106