Back to search

University of Southampton

An investigation into weight fixing networks

Abstract

dc:description.abstract

Deep neural networks require vast computational resources, with data movement between memory and processors dominating energy consumption. This thesis presents novel compression techniques that can reduce these costs by minimising the number of unique parameters in neural networks.<br/><br/>We developed Weight Fixing Networks (WFN), which achieve lossless compression using 56x fewer unique weights and 1.9x lower weight-space entropy than state-of-the-art quantisation approaches. However, this revealed a key limitation: treating all weights based solely on value neglects their position-specific importance.<br/><br/>To address this, we introduced Probabilistic Weight Fixing Networks (PWFN), employing Bayesian neural networks to learn position-specific weight uncertainties. This probabilistic framework enables more intelligent compression decisions, reducing a 5-million parameter transformer to just 296 unique values whilst outperforming state-of-the-art quantisation by 1.6% top-1 accuracy on ImageNet.<br/><br/>Our final contribution, Cluster-On-the-Fly PWFN, integrates clustering directly into training, streamlining the compression process. Unexpectedly, these compressed networks also provide well-calibrated uncertainty estimates, enhancing reliability alongside efficiency.<br/><br/>By focusing on whole-network weight reuse and targeting hardware-friendly values, this work demonstrates that neural networks can be compressed far beyond current limits whilst improving both accuracy and uncertainty quantification. These findings have significant implications for deploying efficient and trustworthy AI on resource-constrained devices.

Degree

thesis:*
Name dc:type.qualificationname
Ph.D.
Level dc:type.qualificationlevel
doctoral
Grantor dc:publisher.institution
University of Southampton
Year dc:date.issued
2025

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Subia-Waud, Christopher
Advisors dc:contributor.advisor
  • Dasmahapatra, Srinandan
  • Weddell, Alex
ORCID person:identifier.orcid

Rights

Language dc:language
English, eng

Chain of custody

source
Harvested from
University of Southampton
Base URL
eprints.soton.ac.uk/cgi/oai2
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
related terms
citation

Subia-Waud, Christopher. An investigation into weight fixing networks. doctoral thesis, University of Southampton, 2025.