Back to search

University of British Columbia

Donsker-Varadhan policy matching for reward-free offline reinforcement learning

Abstract

dc:description

This thesis studies an alternative method for reward-free offline reinforcement learning, investigating how well an agent can learn from a fixed dataset when reward guidance is missing, weak, or unreliable. A common way to handle this problem is behavior regularization, where the learned policy is kept close to the behavior policy that generated the dataset. This thesis studies an actor-to-behavior KL version of this idea using the Donsker–Varadhan representation of KL divergence. In contrast to behavior cloning’s behavior-to-actor KL direction, this actor-to-behavior direction can avoid the action-averaging problem. A reward-augmented alternative is also studied and can be used in settings where reward labeling is meaningful. In controlled settings, our method can recover KL divergence. However, its practical performance depends strongly on witness quality, optimization stability, and dataset structure. Therefore, while the method provides a principled framework for reward-free policy optimization, it does not consistently outperform established benchmark algorithms.

Degree

thesis:*
Name thesis:degree_name
Master of Applied Science - MASc
Level thesis:degree_level
master's
Discipline thesis:degree_discipline
Electrical Engineering
Grantor dc:publisher
University of British Columbia
Year dc:date
2026

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Dao, Quang Anh
Contributors dc:contributor
  • Hossain, Md. Jahangir
  • Cheng, Julian

Rights

dc:rights
Statement dc:rights
  • Attribution-NonCommercial-NoDerivatives 4.0 International
Language dc:language
eng

Identifiers

dc:identifier.*
Handle dc:identifier
http://hdl.handle.net/2429/95730

Chain of custody

source
Harvested from
University of British Columbia
Base URL
circle.library.ubc.ca/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
related terms
citation

Dao, Quang Anh. Donsker-Varadhan policy matching for reward-free offline reinforcement learning. master's thesis, University of British Columbia, 2026. http://hdl.handle.net/2429/95730