Back to search

University of Washington

Beyond Memorization: Evaluating Length-Generalization in Transformer-based Language Models

Abstract

dc:description.abstract

Transformer-based large language models have made substantial progress in the NLP community. However, transformers have trouble with length generalization (i.e., extrapolating on different lengths than seen during training). Recently, (Zhou et al., 2024a) proposed the RASP-Generalization Conjecture to predict what tasks are length-generalizable using a few carefully handcrafted tasks in the mathematical domain. This work examines this conjecture by generating hundreds of synthetic tasks written in the shortest RASP-L programs. Our investigation does not support the conjecture. The tasks written in the shortest RASP-L programs are not length-generalizable. Furthermore, our analysis reveals that some unlength generalizable tasks are due to models not stopping generating. It can be easily fixed by using oracle length during evaluation as suggested in previous literature. Additionally, the analysis rejects induction head as the key factor for failure of length generalization as claimed in the previous findings. Despite our work not providing a precise explanation of transformers’ length generalization capability, we show that previous claims cannot extend to other tasks rather than carefully handcrafted tasks.

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Cheng, Yao-Fei
Advisor dc:contributor.advisor
  • Steinert-Threlkeld, Shane

Subjects

dc:subject × 5

Rights

dc:rights
Statement dc:rights
  • CC BY-NC
Language dc:language.iso
en_US

Identifiers

dc:identifier.*
Handle dc:identifier.uri
https://hdl.handle.net/1773/53677

Chain of custody

source
Harvested from
University of Washington
Base URL
digital.lib.washington.edu/server/oai/request
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Cheng, Yao-Fei. Beyond Memorization: Evaluating Length-Generalization in Transformer-based Language Models. 2025. https://hdl.handle.net/1773/53677