Back to search

Cornell University

Human-in-the-Loop Methods for Robot Failure Recovery

Abstract

dc:description.abstract

Robots operating in unstructured environments inevitably encounter failures, especially in robot caregiving scenarios. For instance, in the domain of robot-assisted bite acquisition, failures can occur because of the variation in the shapes, compliance, sizes, and textures of food items. Because of this variation, existing fully autonomous learning strategies will often fail to generalize to new environments, leading to failures at multiple stages of the robot decision-making pipeline, including perception, planning, and control. While autonomous strategies for failure recovery exist, these tend to be sample inefficient, requiring potentially unsafe interactions with the environment, or require manual human engineering. Human-in-the-loop methods for failure recovery leverage the presence of the human user, who is present during both training and deployment time, to recover from failures. However, excessive or poorly targeted queries may impose unnecessary cognitive and physical workload on the human partner. In this thesis, we develop human-in-the-loop methods for robot failure recovery that incorporate human feedback into the robot decision-making pipeline to recover efficiently from robot failures, while respecting the querying workload imposed on the human. These methods leverage confidence scores along with predictive models of human workload to decide what and when to ask for help. We first develop a human-in-the-loop method for low-level action failure recovery, LinUCB-QG, which decides when to ask for help in a contextual bandit bite acquisition setting. This method leverages a predictive model of human workload, trained on data that we collected in an online user study involving participants both with and without mobility limitations. We then extend the insights from this method to develop a human-in-the-loop framework for modular failure recovery, addressing failure modes across the pipeline. We evaluate both methods through extensive simulations, across both food datasets and controlled synthetic environments, as well as through real-robot user studies, including in-lab studies with users without mobility limitations, and in-home studies with users with mobility limitations. Through our evaluations, we find that our approaches improve recovery success while reducing the workload imposed on users. Additionally, our evaluations highlight how explicitly reasoning about both robot uncertainty and human effort can enable more efficient and user-centered failure recovery in collaborative robots.

Degree

thesis:*
Name thesis:degree_name
Ph. D., Computer Science
Level thesis:degree_level
Doctor of Philosophy
Discipline thesis:degree_discipline
Computer Science
Grantor
Cornell University
Year dc:date.issued
2026

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Banerjee, Rohan
Chair dc:contributor.chair
  • Dean, Sarah
Committee members dc:contributor.committeemember
  • Bhattacharjee, Tapomayukh
  • Sun, Wen
  • Campbell, Mark
  • Mehta, Saurabh

Subjects

dc:subject × 3

Rights

dc:rights
Statement dc:rights
  • Attribution 4.0 International
Language dc:language.iso
en

Identifiers

dc:identifier.*
Dc Identifier Other
Banerjee_cornellgrad_0058F_15468
https://www.proquest.com/LegacyDocView/DISSNUM/32574223

Chain of custody

source
Harvested from
Cornell University
Base URL
ecommons.cornell.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Banerjee, Rohan. Human-in-the-Loop Methods for Robot Failure Recovery. Doctor of Philosophy thesis, Cornell University, 2026. https://hdl.handle.net/1813/126668