Back to search

Cornell University

Bridging the Embodiment Gap for Robot Policy Learning from Human Video Data

Abstract

dc:description.abstract

Human videos offer a scalable source of demonstrations for robot learning, but using them to train robot policies is challenging because they lack robot action labels and are collected from an embodiment with different dynamics, morphology, and contact strategies. As a result, directly imitating human motion can transfer infeasible behaviors to the robot, while ignoring human videos discards a large source of task intent and object-interaction information. This thesis studies how robot policies can learn from human videos by extracting the transferable structure of a demonstration without inheriting execution strategies that are specific to the human body.We explore two complementary approaches to bridging this embodiment gap. First, we present X-Diffusion, a cross-embodiment learning framework for training Diffusion Policies from both robot and retargeted human demonstrations. X-Diffusion treats human actions as noisy counterparts of robot actions and uses a classifier to determine when noised human and robot actions become indistinguishable in the forward diffusion process. Human demonstrations are included only at noise levels where embodiment-specific differences have faded, allowing the policy to learn task-relevant guidance without copying infeasible motions. Across five real-world manipulation tasks, X-Diffusion improves average success rates by 16% over naive co-training and manual data filtering. Second, we present X-Sim, a real-to-sim-to-real framework that avoids direct action imitation by using object motion as the transferable signal. X-Sim reconstructs a photorealistic simulation from RGBD human video, tracks object trajectories to define dense object-centric rewards, trains reinforcement learning policies in simulation, and distills those policies into image-conditioned Diffusion Policies for real-world deployment. To reduce the visual gap between simulation and the real world, X-Sim aligns real and simulated observations during online rollouts. Across five manipulation tasks in two environments, X-Sim improves task progress by 30% over hand-tracking and sim-to-real baselines, matches behavior cloning with 10× less data collection time, and generalizes to test-time changes such as novel viewpoints. Together, these methods show that human videos can support robust robot policy learning when they are used at the right level of abstraction: as noisy action guidance when actions become embodiment-agnostic, or as object-level task specifications when direct action transfer is unreliable.

Degree

thesis:*
Name thesis:degree_name
M.S., Computer Science
Level thesis:degree_level
Master of Science
Discipline thesis:degree_discipline
Computer Science
Grantor
Cornell University
Year dc:date.issued
2026

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Pace, Maximus
Chair dc:contributor.chair
  • Choudhury, Sanjiban
Committee member dc:contributor.committeemember
  • Bershaw, Dwayne

Subjects

dc:subject × 3

Rights

dc:rights
Statement dc:rights
  • Attribution 4.0 International
Language dc:language.iso
en

Identifiers

dc:identifier.*
Dc Identifier Other
Pace_cornell_0058O_12749
https://www.proquest.com/LegacyDocView/DISSNUM/32696236

Chain of custody

source
Harvested from
Cornell University
Base URL
ecommons.cornell.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Pace, Maximus. Bridging the Embodiment Gap for Robot Policy Learning from Human Video Data. Master of Science thesis, Cornell University, 2026. https://hdl.handle.net/1813/126271