Back to search

University of Washington

Talking to Robots: Learning to Ground Human Language in Perception and Execution

Abstract

dc:description.abstract

Advances in computation, sensing, and hardware are enabling robots to perform an increasing variety of tasks in progressively fewer constraints. It is now possible to imagine robots that can operate in traditionally human-centric environments. However, such robots need the flexibility to take instructions and learn about tasks from nonspecialists using language and other natural modalities. At the same time, physically grounded settings provide exciting opportunities for language learning. This thesis describes work on learning to acquire language for human-robot interaction in a physically grounded space. Two use cases are considered: learning to follow route directions through an indoor map, and learning about object attributes from people using unconstrained language and gesture. These problems are challenging because both language and real-world sensing tend to be noisy and ambiguous. This is addressed by reasoning and learning jointly about language and its physical context, parsing into intermediate formal representations that can be interpreted meaningfully by robotic systems. These systems can learn how to follow natural language directions through a map and how to identify objects from human descriptions, even when the underlying concepts are novel to the system, with success rates comparable to or defining the state of the art. Evaluations show that this work takes important steps towards building a robust, flexible, and effective mechanism for bringing together language acquisition and sensing to learn about the world.

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Matuszek, Cynthia
Advisor dc:contributor.advisor
  • Fox, Dieter

Subjects

dc:subject × 4

Rights

dc:rights
Statement dc:rights
  • Copyright is held by the individual authors.
Language dc:language.iso
en_US

Identifiers

dc:identifier.*
Handle dc:identifier.uri
http://hdl.handle.net/1773/27447

Chain of custody

source
Harvested from
University of Washington
Base URL
digital.lib.washington.edu/server/oai/request
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Matuszek, Cynthia. Talking to Robots: Learning to Ground Human Language in Perception and Execution. 2015. http://hdl.handle.net/1773/27447