Back to search

South Dakota State University

A Taxonomy-Driven Modular Defense Against Non-Canonical Language in Vision-Language-Action Models

Abstract

dc:description.abstract

Vision-Language-Action (VLA) models have recently achieved strong performance across manipulation benchmarks, but these benchmarks rely on highly templated instructions on which the models are typically fine-tuned, leaving their behavior under realistic language-side perturbation unclear. It remains an open question whether the linguistic flexibility inherited from vision-language pretraining survives this fine-tuning, or whether the resulting policies become narrowly tuned to benchmark phrasing and brittle to the intent-preserving language variation that real users naturally produce. We address this gap with a systematic study of VLA robustness under non-canonical instructions, comprising three components: a structured taxonomy of intent-preserving variations spanning linguistic, orthographic, and sociolinguistic dimensions; an LLM-based instruction recovery framework that rewrites non-canonical inputs into canonical robot commands as a model-agnostic preprocessing layer; and a proxy-metric framework that selects the best recovery configuration without expensive policy rollouts. The recovery model is fine-tuned on a benchmark-independent corpus generated through taxonomy-guided LLM augmentation, never seeing benchmark-specific instructions during training. Across four VLA models (OpenVLA, π0.5, SmolVLA, FLOWER-VLA) evaluated on LIBERO and CALVIN, we observe sharply different robustness profiles even among models with comparable baseline performance, and find that lightweight recovery restores a substantial portion of the lost performance. Our results indicate that canonical benchmark success does not imply robustness to natural instruction variation, and that recovery is most effective when policies are already partially robust.

Degree

thesis:*
Name thesis:degree_name
Master of Science (MS)
Level thesis:degree_level
Thesis - Open Access
Discipline thesis:degree_discipline
Electrical Engineering and Computer Science

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Samson, Viraj
Contributors dc:contributor
  • Kwanghee Won

Subjects

dc:subject × 1

Rights

dc:rights

Identifiers

dc:identifier.*
Repository record dc:identifier
https://openprairie.sdstate.edu/etd2/2022

Chain of custody

source
Harvested from
South Dakota State University
Base URL
openprairie.sdstate.edu/do/oai/
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Samson, Viraj. A Taxonomy-Driven Modular Defense Against Non-Canonical Language in Vision-Language-Action Models. Thesis - Open Access thesis, https://openprairie.sdstate.edu/etd2/2022