Back to search

University of Southampton

Advancing differentiable program optimisation via novel first and second-order metrics, and adaptive optimisation strategies

Abstract

dc:description.abstract

This thesis is on the optimisation of deep neural networks, how they behave during training, how they can be made more efficient, and how methods and ideas from classical optimisation vary when applied in a deep learning context.<br/><br/>The first part of the thesis is a first-order analysis of the training process of deep neural networks, with a focus on the long range structure of the learning process and how well this can be approximated by a stochastic process.<br/>We show that the learning process of deep neural networks can be reasonably well approximated by a stochastic process, and that this diffusion process can be used to understand the differences between the most popular optimisers including the artefacts from the update equations and the different convergence rates.<br/>The second part of the thesis is on the second-order analysis of the training process of deep neural networks, with a focus on the curvature of the loss surface and how this can be used to improve the optimisation process.<br/>In the third part we propose a new optimisation algorithm, called orthogonalised stochastic gradient descent (OSGD), which is based on the effect of introducing a diversification bias on the convolutional filters via orthonormalisation.<br/><br/>And also show that the adaption from SGD to OSGD can be used to improve the convergence rate of other optimisers, including Adam and RMSProp.<br/><br/>We show that this algorithm can be used to train deep neural networks with fewer epochs and better generalisation performance.<br/><br/>This work concludes with an overview of the results, a discussion of the implications of the work presented in this thesis, and some promising future directions of study.<br/>

Degree

thesis:*
Name dc:type.qualificationname
Ph.D.
Level dc:type.qualificationlevel
doctoral
Grantor dc:publisher.institution
University of Southampton
Year dc:date.issued
2026

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Tuddenham, Mark
Advisors dc:contributor.advisor
  • Prugel-Bennett, Adam
  • Hare, Jonathon

Rights

Language dc:language
English, eng

Chain of custody

source
Harvested from
University of Southampton
Base URL
eprints.soton.ac.uk/cgi/oai2
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
related terms
citation

Tuddenham, Mark. Advancing differentiable program optimisation via novel first and second-order metrics, and adaptive optimisation strategies. doctoral thesis, University of Southampton, 2026.