Back to search

Cornell University

LANGUAGE MODELS FOR ROBOT TASK PLANNING WITH HUMAN DEMONSTRATIONS

Abstract

dc:description.abstract

Two major challenges exist with high-level robot task planning when the goal is under specified: humans have implicit assumptions and preferences they may not articulate when specifying a goal/reward, and visual demonstrations arehard to ground in a form robots can understand. This thesis addresses these challenges leveraging Large Language Models (LLMs) and Vision-Language Models (VLMs) to convert task demonstrations to robot code offline, then adapt to changes planning online. This work is divided into three connected components. Firstly, we developed DEMO2CODE, that generates robot code from demonstrations assumed to be grounded in text form. We focus here on its quality in capturing preferences from real-world demonstrations. Secondly, we delve into grounding visual input in text form with VIDEO2DEMO, focusing on open-vocabulary predicate and action recognition in kitchen tasks. Lastly, we build and deploy an AI task planner to allow collaborative cooking in our MOSAIC framework. We focus here on evaluating the task planner on its safety violations while interacting with participants in our user study. In summary, this thesis focuses on robot task planning and human-robot interaction by using demonstrations for effective code generation, grounding visual information in text, capturing and adhering to preferences of the human, all using generative language models.

Degree

thesis:*
Name thesis:degree_name
M.S., Computer Science
Level thesis:degree_level
Master of Science
Discipline thesis:degree_discipline
Computer Science
Grantor
Cornell University
Year dc:date.issued
2024

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Sharma, Yash
Chair dc:contributor.chair
  • Choudhury, Sanjiban
Committee member dc:contributor.committeemember
  • Edelman, Shimon

Subjects

dc:subject × 4

Rights

dc:rights
Statement dc:rights
  • Attribution-ShareAlike 4.0 International
Language dc:language.iso
en

Identifiers

dc:identifier.*
Dc Identifier Bibid
16575485
Dc Identifier Other
ProQuest Submission ID: 12081
ProQuest Publication ID: 31242170

Chain of custody

source
Harvested from
Cornell University
Base URL
ecommons.cornell.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Sharma, Yash. LANGUAGE MODELS FOR ROBOT TASK PLANNING WITH HUMAN DEMONSTRATIONS. Master of Science thesis, Cornell University, 2024. https://hdl.handle.net/1813/115862