Cornell University
Generalizable Learning for Natural Language Instruction Following on Physical Robots
Abstract
dc:description.abstractRobot applications in unstructured human-inhabited environments, such as households, assistive scenarios, and collaborative industrial tasks, require a human-robot interaction interface through which humans can instruct robots what to do.Natural language is an accessible, expressive, effective, and efficient interface to specify a wide array of instructions and goals to robots. This thesis presents a modular and interpretable representation learning approach to following natural language instructions on physical robots by mapping raw visual observations to continuous control. The approach reconsiders the full robotics stack, including solving perception, planning, mapping, and control challenges, all with a focus on enabling behavior specification in natural language. The presented approach achieved the first demonstration of a physical robot that could follow natural language instructions by mapping RGB images to continuous control. The thesis also introduces a hierarchical approach to following mobile manipulation instructions, showing promising early results on a virtual mobile manipulation benchmark.
Degree
thesis:*- Name thesis:degree_name
- Ph. D., Computer Science
- Level thesis:degree_level
- Doctor of Philosophy
- Discipline thesis:degree_discipline
- Computer Science
- Grantor
- Cornell University
- Year dc:date.issued
- 2021
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Blukis, Valts
- Chair dc:contributor.chair
-
- Artzi, Yoav
- Committee members dc:contributor.committeemember
-
- Snavely, Noah
- Ju, Wendy Guang-wen
- Lee, Daniel Dongyuel
Subjects
dc:subject × 6Rights
dc:rights- Statement dc:rights
-
- Attribution-NonCommercial 4.0 International
- Licence dc:rights.uri
- Language dc:language.iso
- en
Identifiers
dc:identifier.*- Dc Identifier Bibid
- 15312673
- Dc Identifier Other
-
ProQuest Submission ID: 12901
ProQuest Publication ID: 28864494