Back to search

University of Washington

mSCAN - a Multilingual Dataset for Compositional Generalization Evaluation

Abstract

dc:description.abstract

Language models achieve remarkable results on a variety of tasks, yet still struggle on compositional generalization benchmarks. The majority of these benchmarks evaluate performance in English only, leaving open the question of whether these results generalize to other languages. As an initial step to answering this question, we introduce mSCAN, a multilingual adaptation of the SCAN dataset covering Mandarin Chinese, French, Hindi and Russian. It was produced by a rule-based translation, developed in cooperation with native speakers. We then showcase this dataset on some in-context learning experiments on multiple open-source multilingual models.

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Reymond, Amélie Thu Tâm
Advisor dc:contributor.advisor
  • Steinert-Threlkeld, Shane

Subjects

dc:subject × 6

Rights

dc:rights
Statement dc:rights
  • none
Language dc:language.iso
en_US

Identifiers

dc:identifier.*
Handle dc:identifier.uri
https://hdl.handle.net/1773/53674

Chain of custody

source
Harvested from
University of Washington
Base URL
digital.lib.washington.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Reymond, Amélie Thu Tâm. mSCAN - a Multilingual Dataset for Compositional Generalization Evaluation. 2025. https://hdl.handle.net/1773/53674