Back to search

University of British Columbia

Critic-derived shaping in Proximal Policy Optimization

Abstract

dc:description

The simplicity and stability of Proximal Policy Optimization (PPO) in reinforcement learning make it popular, yet its on-policy training loop may be sample-inefficient. This thesis investigates whether PPO can leverage temporal-difference (TD) error, a signal that it already computes, in a regulated manner. We introduce critic-derived advantage shaping (CDAS), a method which employs normalized positive TD error as an actor-side shaping signal, and critic-derived value shaping (CDVS), which applies the same detached, normalized bonus to the critic target. While CDAS-0 and CDVS-0 employ one-step positive TD error, we test additional variants called CDAS-λ and CDVS-λ, which propagate future positive critic surprise backward via a GAE-style bonus trace. We find that PPO can benefit from positive critic surprise in advantage-side shaping, and especially critic-side shaping, but only under strict scale control. When the bonus is modest in comparison to the extrinsic advantage, actor-side CDAS improves performance on several MuJoCo tasks and performs particularly well on the Atari game Gravitar. Conservative CDAS-0 and CDAS-λ settings in MuJoCo increase mean PPO-normalized final return by around 4–5% above PPO. However, the improvements are environment-dependent; Venture demonstrates how visually prominent but task-misaligned transitions can draw the TD-error signal and impair learning, while Hopper performance deteriorates when the coefficient is too high. Critic-side shaping is stronger in aggregate—CDVS-0 improves mean PPO-normalized final return by 7.4% across the studied MuJoCo environments, and the strongest critic-only CDVS-λ configuration improves it by 12.5%. In the Atari critic-side sweep, the strongest aggregate CDVS-λ configuration improves mean final return by 14.2% and mean AUC by about 16%, with AUC gains in all five Atari games. Overall, critic-derived shaping is useful until it overpowers PPO's extrinsic learning structure: TD-error-based bonuses must stay secondary to PPO's extrinsic learning signals in order to be beneficial.

Degree

thesis:*
Name thesis:degree_name
Master of Science - MSc
Level thesis:degree_level
master's
Discipline thesis:degree_discipline
Computer Science
Grantor dc:publisher
University of British Columbia
Year dc:date
2026

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Deaconu, George Alexandru
Contributors dc:contributor
  • Irani, Pourang
  • Milani, Abbas S.

Rights

dc:rights
Statement dc:rights
  • Attribution-NonCommercial-NoDerivatives 4.0 International
Language dc:language
eng

Identifiers

dc:identifier.*
Handle dc:identifier
http://hdl.handle.net/2429/95164

Chain of custody

source
Harvested from
University of British Columbia
Base URL
circle.library.ubc.ca/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
related terms
citation

Deaconu, George Alexandru. Critic-derived shaping in Proximal Policy Optimization. master's thesis, University of British Columbia, 2026. http://hdl.handle.net/2429/95164