Back to search

University of Leeds

Seeing to learn: Observational learning of robotic manipulation tasks

Abstract

dc:description.abstract

Learning new tasks has always been a challenging problem in robotics. Even though several approaches have been proposed, from manual programming to learning from demonstrations, the field has directions which require further research and development. This thesis focuses on one of these relatively unexplored areas: observational learning. We present O2A, a novel method for learning to perform robotic manipulation tasks from a single (one-shot) third-person demonstration video. The key novelty lies in pre-training a feature extractor for creating an abstract feature representation for actions that we call ‘action vectors’. The action vectors are extracted using a 3D-CNN network pre-trained for action recognition on a generic action dataset. The distance between the action vectors from the observed third-person demonstration and trial robot executions are used as a reward/cost for learning of the demonstrated task. We report on experiments in simulation and on a real robot, with changes in viewpoint of observation, properties of the objects involved, scene background and morphology of the manipulator between the demonstration and the learning domains. O2A outperforms baseline approaches under different domain shifts and has comparable performance with an oracle (that uses an ideal reward function). We also plot visualisation of trajectories and show that our method has high reward for desired trajectories. Finally, we present a framework for extending observational learning with multi modal observations. We report our initial experiments and results in the future works.

Degree

thesis:*
Name dc:type.qualificationname
Ph.D
Level dc:type.qualificationlevel
doctoral
Grantor dc:publisher.institution
University of Leeds
Year dc:date.issued
2021

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Pauly, Leo
Advisors dc:contributor.advisor
  • Fuentes, Raul
  • Hogg, David

Rights

Language dc:language.dcterms_iso639_2
eng

Identifiers

dc:identifier.*
Identifier
uk.bl.ethos.834050

Chain of custody

source
Harvested from
White Rose University Consortium
Base URL
etheses.whiterose.ac.uk/cgi/oai2
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
related terms
citation

Pauly, Leo. Seeing to learn: Observational learning of robotic manipulation tasks. doctoral thesis, University of Leeds, 2021.