Back to search

Cornell University

Learning Robot Manipulation Skills from Human Videos

Abstract

dc:description.abstract

Teaching robots to manipulate objects requires large amounts of demonstrated behavior, yet collecting expert robot demonstrations is expensive, slow, and difficult to scale. Human videos offer an abundant alternative — they are cheap to record, capture a rich diversity of manipulation strategies, and require no robot involvement. However, directly leveraging human videos for robot policy training poses a fundamental challenge: humans and robots differ significantly in embodiment, making human actions difficult or impossible to replicate on a robot. This thesis presents two complementary frameworks that address this cross-embodiment gap. The first contribution, X-Sim, sidesteps execution differences entirely by avoiding action retargeting. Rather than extracting robot actions from human hand motion, X-Sim identifies object motion — observable in any RGBD video — as a dense, embodiment- agnostic supervisory signal. A photorealistic simulation is reconstructed from the human video, and tracked object trajectories define object-centric rewards for training reinforcement learning policies in simulation. These policies are distilled into image- conditioned diffusion policies and transferred to the real world via an online domain adaptation technique that aligns simulated and real observations at test time, requiring zero robot teleoperation data. Across five manipulation tasks, X-Sim improves task progress by 30% over hand-tracking baselines and achieves comparable performance to behavior cloning with 10× less data collection time. The second contribution, X-Diffusion, takes an alternate approach that avoids the complexity of simulation and RL by incorporating human demonstrations directly into imitation learning. It treats human hand actions as noisy counterparts of robot actions, grounded in the observation that as Gaussian noise is added during the forward diffusion process, low-level embodiment-specific differences fade while high-level task structure is preserved. X-Diffusion trains a classifier to identify, for each human action, the minimum noise level at which it becomes indistinguishable from a robot action, and selectively integrates human demonstrations into diffusion policy training only beyond this threshold. This adaptive co-training avoids learning kinematically infeasible be- haviors while still extracting useful guidance from the full human dataset, improving average task success by 16% over naive co-training and manual data filtering across five real-world manipulation tasks. Together, these works demonstrate two principled strategies for bridging the human– robot embodiment gap — one through simulation as an intermediary, and one through the structure of the diffusion process itself — establishing human video as a viable and scalable source of supervision for robot manipulation learning.

Degree

thesis:*
Name thesis:degree_name
M.S., Computer Science
Level thesis:degree_level
Master of Science
Discipline thesis:degree_discipline
Computer Science
Grantor
Cornell University
Year dc:date.issued
2026

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Dan, Prithwish
Chair dc:contributor.chair
  • Choudhury, Sanjiban
Committee members dc:contributor.committeemember
  • Ma, Wei-Chiu
  • Bhattacharjee, Tapomayukh

Subjects

dc:subject × 3

Rights

dc:rights
Statement dc:rights
  • Attribution 4.0 International
Language dc:language.iso
en

Identifiers

dc:identifier.*
Dc Identifier Other
Dan_cornell_0058O_12686
https://www.proquest.com/LegacyDocView/DISSNUM/32675630

Chain of custody

source
Harvested from
Cornell University
Base URL
ecommons.cornell.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Dan, Prithwish. Learning Robot Manipulation Skills from Human Videos. Master of Science thesis, Cornell University, 2026. https://hdl.handle.net/1813/126248