University of British Columbia
Critic-derived shaping in Proximal Policy Optimization
Abstract
dc:descriptionThe simplicity and stability of Proximal Policy Optimization (PPO) in reinforcement learning make it popular, yet its on-policy training loop may be sample-inefficient. This thesis investigates whether PPO can leverage temporal-difference (TD) error, a signal that it already computes, in a regulated manner. We introduce critic-derived advantage shaping (CDAS), a method which employs normalized positive TD error as an actor-side shaping signal, and critic-derived value shaping (CDVS), which applies the same detached, normalized bonus to the critic target. While CDAS-0 and CDVS-0 employ one-step positive TD error, we test additional variants called CDAS-λ and CDVS-λ, which propagate future positive critic surprise backward via a GAE-style bonus trace. We find that PPO can benefit from positive critic surprise in advantage-side shaping, and especially critic-side shaping, but only under strict scale control. When the bonus is modest in comparison to the extrinsic advantage, actor-side CDAS improves performance on several MuJoCo tasks and performs particularly well on the Atari game Gravitar. Conservative CDAS-0 and CDAS-λ settings in MuJoCo increase mean PPO-normalized final return by around 4–5% above PPO. However, the improvements are environment-dependent; Venture demonstrates how visually prominent but task-misaligned transitions can draw the TD-error signal and impair learning, while Hopper performance deteriorates when the coefficient is too high. Critic-side shaping is stronger in aggregate—CDVS-0 improves mean PPO-normalized final return by 7.4% across the studied MuJoCo environments, and the strongest critic-only CDVS-λ configuration improves it by 12.5%. In the Atari critic-side sweep, the strongest aggregate CDVS-λ configuration improves mean final return by 14.2% and mean AUC by about 16%, with AUC gains in all five Atari games. Overall, critic-derived shaping is useful until it overpowers PPO's extrinsic learning structure: TD-error-based bonuses must stay secondary to PPO's extrinsic learning signals in order to be beneficial.
Degree
thesis:*- Name thesis:degree_name
- Master of Science - MSc
- Level thesis:degree_level
- master's
- Discipline thesis:degree_discipline
- Computer Science
- Grantor dc:publisher
- University of British Columbia
- Year dc:date
- 2026
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Deaconu, George Alexandru
- Contributors dc:contributor
-
- Irani, Pourang
- Milani, Abbas S.
Rights
dc:rights- Statement dc:rights
-
- Attribution-NonCommercial-NoDerivatives 4.0 International
- Language dc:language
- eng
Identifiers
dc:identifier.*- Handle dc:identifier
- http://hdl.handle.net/2429/95164