University of Washington
mSCAN - a Multilingual Dataset for Compositional Generalization Evaluation
Abstract
dc:description.abstractLanguage models achieve remarkable results on a variety of tasks, yet still struggle on compositional generalization benchmarks. The majority of these benchmarks evaluate performance in English only, leaving open the question of whether these results generalize to other languages. As an initial step to answering this question, we introduce mSCAN, a multilingual adaptation of the SCAN dataset covering Mandarin Chinese, French, Hindi and Russian. It was produced by a rule-based translation, developed in cooperation with native speakers. We then showcase this dataset on some in-context learning experiments on multiple open-source multilingual models.
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Reymond, Amélie Thu Tâm
- Advisor dc:contributor.advisor
-
- Steinert-Threlkeld, Shane
Subjects
dc:subject × 6Rights
dc:rights- Statement dc:rights
-
- none
- Language dc:language.iso
- en_US
Identifiers
dc:identifier.*- Handle dc:identifier.uri
- https://hdl.handle.net/1773/53674