Back to search

Cornell University

Principled Off-Policy Imitation Learning via Boosting

Abstract

dc:description.abstract

Imitation learning is a promising paradigm to learn policies to solve a variety of tasks given some expert data. In particular, off-policy imitation learning is particularly nice for practitioners, as it in principle allows the policy to use previously collected data to improve, similar to standard value-based off-policy reinforcement learning algorithms such as Deep Q Learning and actor-critic methods. However, this is generally ill-defined, as the policy improvement operator can generally only in principle be applied to data collected by the most recent policy, making the algorithm on-policy. To mitigate this while still remaining off-policy, we design an actor-critic method where we treat the replay buffer as a collection of data from a set of weak learners. Our algorithm more appropriately weights each weak learner’s data when it comes to sampling for policy optimization, offering a principled way to mitigate the above distribution mismatch problem in the off-policy setting. We apply this technique to both state and vision-based tasks in the DeepMind Control Suite domain and see that our method does indeed improve learning in terms of sample efficiency.

Degree

thesis:*
Name thesis:degree_name
M.S., Computer Science
Level thesis:degree_level
Master of Science
Discipline thesis:degree_discipline
Computer Science
Grantor
Cornell University
Year dc:date.issued
2023

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Sreenivas, Dhruv
Chair dc:contributor.chair
  • Sun, Wen
Committee member dc:contributor.committeemember
  • Kleinberg, Robert

Subjects

dc:subject × 3

Rights

Language dc:language.iso
en

Identifiers

dc:identifier.*
Dc Identifier Bibid
16219185
Dc Identifier Other
ProQuest Submission ID: 11870
ProQuest Publication ID: 30573370

Chain of custody

source
Harvested from
Cornell University
Base URL
ecommons.cornell.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Sreenivas, Dhruv. Principled Off-Policy Imitation Learning via Boosting. Master of Science thesis, Cornell University, 2023. https://hdl.handle.net/1813/114489