Cornell University
Linear recurrent models for robust and interpretable long-context modeling
Abstract
dc:description.abstractTransformers, powered by self-attention, have become foundational across modalities due to their scalability through parallelism. However, many real-world tasks—from byte-level language modeling to retrieval and reasoning—demand processing extended contexts, often spanning tens of thousands of tokens. The quadratic complexity of Transformers during training and linear KV-cache growth at inference makes them computationally prohibitive for such tasks, necessitating the exploration of more efficient architectures. Linear-time recurrent models and hybrid architectures, which combine linear recurrence with local attention, have emerged as promising alternatives to address these challenges. This thesis investigates two interconnected dimensions of long-context modeling: the development of scalable architectures and the interpretability of their underlying recurrence mechanisms. In the context of scalable architectures, we adapt the Mamba linear recurrent byte-level sequence modeling, tackling the computational challenges posed by extended sequence lengths. By introducing speculative decoding that combines subword drafting with byte-level verification, we demonstrate that token-free models can achieve subword-like inference speedups while maintaining robustness to noise and generalization over long sequences. On the interpretability front, we focus on hybrid models like RecurrentGemma, where recurrent hidden states serve as memory to bridge local attention windows. Using sparse autoencoders, we extract interpretable features from these recurrent states to explore their role in retaining long-range information. Through tasks such as long-context passkey retrieval, we reveal how these mechanisms facilitate extended context dependencies and offer insights into the memory of these fixed-memory models. Through these contributions, this thesis advances the understanding and development of architectures that combine scalability and interpretability for long-context modeling.
Degree
thesis:*- Name thesis:degree_name
- M.S., Computer Science
- Level thesis:degree_level
- Master of Science
- Discipline thesis:degree_discipline
- Computer Science
- Grantor
- Cornell University
- Year dc:date.issued
- 2024
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Gangavarapu, Tushaar
- Chair dc:contributor.chair
-
- Rush, Alexander
- Committee member dc:contributor.committeemember
-
- Bindel, David
Subjects
dc:subject × 5Rights
dc:rights- Statement dc:rights
-
- Attribution 4.0 International
- Licence dc:rights.uri
- Language dc:language.iso
- en
Identifiers
dc:identifier.*- Dc Identifier Bibid
- 16921940
- Dc Identifier Other
-
ProQuest Submission ID: 12319
ProQuest Publication ID: 31641441