University of British Columbia
Donsker-Varadhan policy matching for reward-free offline reinforcement learning
Abstract
dc:descriptionThis thesis studies an alternative method for reward-free offline reinforcement learning, investigating how well an agent can learn from a fixed dataset when reward guidance is missing, weak, or unreliable. A common way to handle this problem is behavior regularization, where the learned policy is kept close to the behavior policy that generated the dataset. This thesis studies an actor-to-behavior KL version of this idea using the Donsker–Varadhan representation of KL divergence. In contrast to behavior cloning’s behavior-to-actor KL direction, this actor-to-behavior direction can avoid the action-averaging problem. A reward-augmented alternative is also studied and can be used in settings where reward labeling is meaningful. In controlled settings, our method can recover KL divergence. However, its practical performance depends strongly on witness quality, optimization stability, and dataset structure. Therefore, while the method provides a principled framework for reward-free policy optimization, it does not consistently outperform established benchmark algorithms.
Degree
thesis:*- Name thesis:degree_name
- Master of Applied Science - MASc
- Level thesis:degree_level
- master's
- Discipline thesis:degree_discipline
- Electrical Engineering
- Grantor dc:publisher
- University of British Columbia
- Year dc:date
- 2026
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Dao, Quang Anh
- Contributors dc:contributor
-
- Hossain, Md. Jahangir
- Cheng, Julian
Rights
dc:rights- Statement dc:rights
-
- Attribution-NonCommercial-NoDerivatives 4.0 International
- Language dc:language
- eng
Identifiers
dc:identifier.*- Handle dc:identifier
- http://hdl.handle.net/2429/95730