Back to search

University of British Columbia

Learning dynamics of deep learning -- force analysis of deep neural networks

Abstract

dc:description

This thesis investigates the learning dynamics of deep learning systems through a local, physics-inspired analytical lens. Motivated by the need for fine-grained insights into model behavior, we begin with the step-wise influence that a single training example exerts on a specific observing example during learning. Central to our approach is the proposed AKG decomposition, which dissects this influence into three interpretable components: similarity (K), normalization (A), and prediction gap (G). This decomposition enables an analogy with classical force analysis: the force originates from G, is shaped by AK, and is ultimately applied to the target object, e.g., to the model confidence, output, hidden representations, or parameters. Building upon this foundation, we gradually scale the analysis from individual interactions to cumulative effects over time, akin to tracking an object’s motion under multiple forces. We apply it to the following problems. Supervised classification: We study the learning trajectories of examples with varying difficulty and reveal an interesting "zig-zag" pattern that emerges during optimization. Our analysis explains this behavior and inspires a novel knowledge distillation method, Filter-KD, which improves supervision signals for student models. Large language model (LLM) finetuning: We extend the framework to account for the autoregressive nature of LLMs and the presence of negative gradients. The unified perspective explains behaviors across finetuning methods such as SFT, DPO, and GRPO. We also highlight the critical role of negative gradients. In particular, we identify the "squeezing effect": a counterintuitive phenomenon caused by improperly applied gradient ascent. Representation learning: We explore the dynamics of hidden features, revealing how adaptation energy and directions influence the feature drift. Our analysis leads to a provable pattern of feature adaptation in a head-probing then finetuning pipeline, offering insights and inspiring several practical strategies. Simplicity bias and compositional learning: Revisiting foundational questions about why structured representations are learned faster, we apply our framework to a compositional learning setting. Our findings align with principles such as Occam’s Razor and the idea of "compression for AGI," offering a novel dynamical explanation rooted in compression and learning speed.

Degree

thesis:*
Name thesis:degree_name
Doctor of Philosophy - PhD
Level thesis:degree_level
doctoral
Discipline thesis:degree_discipline
Computer Science
Grantor dc:publisher
University of British Columbia
Year dc:date
2025

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Ren, Yi (Joshua)
Contributors dc:contributor
  • Sutherland, Danica J.

Rights

dc:rights
Statement dc:rights
  • Attribution-NonCommercial-NoDerivatives 4.0 International
Language dc:language
eng

Identifiers

dc:identifier.*
Handle dc:identifier
http://hdl.handle.net/2429/92432

Chain of custody

source
Harvested from
University of British Columbia
Base URL
circle.library.ubc.ca/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
related terms
citation

Ren, Yi (Joshua). Learning dynamics of deep learning -- force analysis of deep neural networks. doctoral thesis, University of British Columbia, 2025. http://hdl.handle.net/2429/92432