Back to search

University of Southampton

Pruning resnet neural networks block by block

Abstract

dc:description.abstract

Neural network pruning has gained popularity for deep models with the goal of reducing storage and computational requirements, both for inference and training. Pruning weights from neural networks to make models sparse has been discussed for a long time starting with in 1990s. Structured pruning is more recent and it aims to minimize the size of neural networks as opposed to only introducing sparsity, having the advantage of not requiring specialized hardware or software. Most structured pruning works focus on neurons or convolutional filters. ResNet is a model architecture composed of many blocks that are linked with residual connections. In our work, we explore pruning larger structures – ResNet blocks, and thoroughly study the feasibility of block by block pruning whilst keeping track of the cost of fine-tuning the pruned networks. We use different block saliency metrics as well as different fine-tuning schedules and parameters. Pruning 27 blocks (50%) from a ResNet-110, in our best configuration, gives 6.48% test error on CIFAR-10, a 0.45% loss from the initial model. When pruning 45 blocks to obtain a similar size to that of a ResNet-20, our best method has a 1.92% loss from initial, 7.95% error. We observe that training small, standard ResNet configurations gives better results than pruning and argue that pruning block by block is only effective for pruning a small number of blocks, or when starting with a model pre-trained elsewhere and the cost of fine-tuning is of concern (fine-tuning alone can be much cheaper than training from scratch and give acceptable results, depending on the pruning configuration). Finally, our pruning work is produced from hundreds of experiments, and a by-product of running, organising and analysing them is our experiment management framework, dbx, aims to simplify this process by storing experiments and their results (as logs) in repositories that can be queried and synchronized between computers.

Degree

thesis:*
Name dc:type.qualificationname
Ph.D.
Level dc:type.qualificationlevel
doctoral
Grantor dc:publisher.institution
University of Southampton
Year dc:date.issued
2022

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Velici, Vlad, Sebastian
Advisor dc:contributor.advisor
  • Prugel-Bennett, Adam
ORCID person:identifier.orcid

Rights

Language dc:language
English, eng

Chain of custody

source
Harvested from
University of Southampton
Base URL
eprints.soton.ac.uk/cgi/oai2
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
related terms
citation

Velici, Vlad, Sebastian. Pruning resnet neural networks block by block. doctoral thesis, University of Southampton, 2022.