Back to search

Georgia Institute of Technology

A multimodal execution monitor for assistive robots

Abstract

dc:description.abstract

Assistive robots have the potential to serve as caregivers, providing assistance with activities of daily living to people with disabilities. Monitoring when something has gone wrong could help assistive robots operate more safely and effectively around people. However, the complexity of interacting with people and objects in human environments can make challenges in monitoring operations. By monitoring multimodal sensory signals, an execution monitor could perform a variety of roles, such as detecting success, determining when to switch behaviors, and otherwise exhibiting more common sense. The purpose of this dissertation is to introduce a multimodal execution monitor to improve safety and success of assistive manipulation services. To accomplish this goal, we make three main contributions. First, we introduce a data-driven anomaly detector, a part of the monitor, that reports anomalous task executions from multimodal sensory signals online. Second, we introduce a data-driven anomaly classifier that recognizes the type and cause of common anomalies through an artificial neural network after fusing multimodal features. Lastly, as the main testbed of the monitoring system, we introduce a robot-assisted feeding system for people with disabilities, using a general-purpose mobile manipulator (a PR2 robot). We evaluate the monitoring system with haptic, visual, auditory, and kinematic sensing during household tasks and human-robot interactive tasks including feeding assistance. We show multimodality improves the performance of monitoring methods by detecting and classifying a broader range of anomalies. Overall, our research demonstrates the multimodal execution monitoring system helps the assistive manipulation system to provide safe and successful assistance for people with disabilities.

Degree

thesis:*
Level thesis:degree_level
Doctoral
Department dc:contributor.department
Interactive Computing
Grantor dc:publisher
Georgia Institute of Technology
Year dc:date.issued
2018

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Park, Daehyung
Advisor dc:contributor.advisor
  • Kemp, Charles C.
Committee members dc:contributor.committeemember
  • Boots, Byron
  • Chernova, Sonia
  • Rehg, James M.
  • Trumbower, Randy

Subjects

dc:subject × 2

Rights

Language dc:language.iso
en_US

Identifiers

dc:identifier.*
Handle dc:identifier.uri
http://hdl.handle.net/1853/59860

Chain of custody

source
Harvested from
Georgia Tech
Base URL
repository.gatech.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Park, Daehyung. A multimodal execution monitor for assistive robots. Doctoral thesis, Georgia Institute of Technology, 2018. http://hdl.handle.net/1853/59860