Back to search

Cornell University

Holistic Scene Understanding for Real-Time Human-Autonomy Systems Interaction and Teaming

Abstract

dc:description.abstract

Effective human-autonomy collaboration requires sophisticated testing environments and intelligent perception systems that respond in real time to dynamic situations. As autonomous robots become increasingly prevalent in society, their ability to understand complex scenes and interact naturally with human teammates becomes critical for applications ranging from search and rescue to healthcare and manufacturing. Current approaches face significant limitations: field testing lacks detailed ground truth for algorithm validation, while conventional perception systems operate without close coupling to the vehicle's motor control, introducing latency that hinders fluid interaction within human-autonomy teams. This dissertation addresses these challenges through two complementary research thrusts. The first contribution introduces the Real-Time Human Autonomous Systems Collaborations (RealTHASC) facility, a cyber-physical extended reality (XR) testbed that seamlessly interfaces real and virtual agents with photorealistic simulated environments. By integrating motion capture, wearable sensors, and virtual reality technologies, RealTHASC creates a unified framework where physical robots and humans interact with virtual agents through real-time feedback. Human subjects and robots in the laboratory space are transferred into synthetic environments as avatars that experience the virtual world through simulated sensors while maintaining kinematic coupling with their real counterparts. The second contribution presents a perception-in-action framework that enables robot-embodied visual understanding for real-time human-robot interaction. This event-driven approach integrates motor control and perception in a closed-loop system, allowing mobile robots to interact with nearby people using semantic labels and hand gestures while navigating complex environments. By coupling vision and feedback control designs, the system enables robots to respond to visual scenes through dynamic and autonomous functionalities, overcoming limitations of traditional computer vision techniques that rely on passive, open-loop processing. The framework exploits controlled camera motion to actively simplify visual perception tasks, creating a synergistic relationship between perception and action that enhances efficiency and responsiveness. Experimental results demonstrate effectiveness across diverse applications, with RealTHASC validated through case studies of mixed real/virtual interactions and the perception-in-action framework proving robust in challenging environments. Together, these advances provide a foundation for holistic scene understanding that facilitates natural, effective collaboration between humans and autonomous systems in complex scenarios.

Degree

thesis:*
Name thesis:degree_name
Ph. D., Mechanical Engineering
Level thesis:degree_level
Doctor of Philosophy
Discipline thesis:degree_discipline
Mechanical Engineering
Grantor
Cornell University
Year dc:date.issued
2025

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Paradise, Andre
Chair dc:contributor.chair
  • Ferrari, Silvia
Committee members dc:contributor.committeemember
  • Petersen, Kirstin
  • Albertson, John

Subjects

dc:subject × 6

Rights

dc:rights
Statement dc:rights
  • Attribution 4.0 International
Language dc:language.iso
en

Identifiers

dc:identifier.*
Dc Identifier Bibid
17423416
Dc Identifier Other
ProQuest Submission ID: 15323
ProQuest Publication ID: 32280853

Chain of custody

source
Harvested from
Cornell University
Base URL
ecommons.cornell.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Paradise, Andre. Holistic Scene Understanding for Real-Time Human-Autonomy Systems Interaction and Teaming. Doctor of Philosophy thesis, Cornell University, 2025. https://hdl.handle.net/1813/121033