Back to search

University of British Columbia

Theory and practice of regularized neural ordinary differential equations

Abstract

dc:description

Deep Neural Networks (DNNs) have achieved remarkable success in various machine learning domains, but their training remains computationally intensive and susceptible to architectural choices. Residual Networks (ResNets) mitigate some of these problems. However, they still face challenges with rapid weight and bias changes between layers, leading to unstable training, slow convergence, and over-fitting, especially with limited data. This thesis addresses these limitations by exploring Neural Ordinary Differential Equation (NODE) model architecture, which offer a continuous perspective on network depth. We propose and investigate a regularization method that promotes smoother transitions between layers of network parameters. Our work presents both theoretical foundations and empirical validations. We derive the Euler-Lagrange equations to characterize the optimal continuum limit of regularized NODEs and analyze the asymptotic behaviour of network parameters under strong regularization. Experimentally, we conduct a comparison between regularized NODEs and standard ResNets on image classification tasks using the MNIST and CIFAR-10 datasets. Our findings demonstrate that regularized NODEs, particularly when combined with a soft-restart training strategy that progressively decreases the regularization parameter, exhibit superior training stability, faster convergence with limited data, and increased robustness to varying network depths compared to ResNets. We demonstrate the usage of regularized NODEs on real life biomedical data and compare to existing models. This research highlights the advantages of regularized NODEs as a more stable and efficient architecture for deep learning, offering significant benefits in scenarios with limited computational resources or scarce data.

Degree

thesis:*
Name thesis:degree_name
Master of Science - MSc
Level thesis:degree_level
master's
Discipline thesis:degree_discipline
Mathematics
Grantor dc:publisher
University of British Columbia
Year dc:date
2025

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Chang, Kwok Ho Hody
Contributors dc:contributor
  • Wetton, Brian
  • Gopaluni, Bhushan

Rights

dc:rights
Statement dc:rights
  • Attribution-NonCommercial-NoDerivatives 4.0 International
Language dc:language
eng

Identifiers

dc:identifier.*
Handle dc:identifier
http://hdl.handle.net/2429/92216

Chain of custody

source
Harvested from
University of British Columbia
Base URL
circle.library.ubc.ca/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
related terms
citation

Chang, Kwok Ho Hody. Theory and practice of regularized neural ordinary differential equations. master's thesis, University of British Columbia, 2025. http://hdl.handle.net/2429/92216