Back to search

Cornell University

Generalizable Learning for Natural Language Instruction Following on Physical Robots

Abstract

dc:description.abstract

Robot applications in unstructured human-inhabited environments, such as households, assistive scenarios, and collaborative industrial tasks, require a human-robot interaction interface through which humans can instruct robots what to do.Natural language is an accessible, expressive, effective, and efficient interface to specify a wide array of instructions and goals to robots. This thesis presents a modular and interpretable representation learning approach to following natural language instructions on physical robots by mapping raw visual observations to continuous control. The approach reconsiders the full robotics stack, including solving perception, planning, mapping, and control challenges, all with a focus on enabling behavior specification in natural language. The presented approach achieved the first demonstration of a physical robot that could follow natural language instructions by mapping RGB images to continuous control. The thesis also introduces a hierarchical approach to following mobile manipulation instructions, showing promising early results on a virtual mobile manipulation benchmark.

Degree

thesis:*
Name thesis:degree_name
Ph. D., Computer Science
Level thesis:degree_level
Doctor of Philosophy
Discipline thesis:degree_discipline
Computer Science
Grantor
Cornell University
Year dc:date.issued
2021

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Blukis, Valts
Chair dc:contributor.chair
  • Artzi, Yoav
Committee members dc:contributor.committeemember
  • Snavely, Noah
  • Ju, Wendy Guang-wen
  • Lee, Daniel Dongyuel

Subjects

dc:subject × 6

Rights

dc:rights
Statement dc:rights
  • Attribution-NonCommercial 4.0 International
Language dc:language.iso
en

Identifiers

dc:identifier.*
Dc Identifier Bibid
15312673
Dc Identifier Other
ProQuest Submission ID: 12901
ProQuest Publication ID: 28864494

Chain of custody

source
Harvested from
Cornell University
Base URL
ecommons.cornell.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Blukis, Valts. Generalizable Learning for Natural Language Instruction Following on Physical Robots. Doctor of Philosophy thesis, Cornell University, 2021. https://hdl.handle.net/1813/110811