Back to search

Georgia Institute of Technology

DES: Delayed Eps-Shrinking for Faster Once-For-All Training

Abstract

dc:description.abstract

CNNs are increasingly deployed across different hardware, dynamic environments, and low-power embedded devices. This has led to the design and training of CNN architectures with the goal of maximizing accuracy subject to such variable deployment constraints. As the number of deployment scenarios grows, there is a need to find scalable solutions to design and train specialized CNNs. Once-for-all training has emerged as a scalable approach that jointly co-trains many models (subnets) at once with a constant training cost and finds specialized CNNs later. The scalability is achieved by training the full model and simultaneously reducing it to smaller subnets that share model weights (weight-sharing). However, existing once-for-all training approaches incur huge training costs reaching 1200 GPU hours. We argue this is because they either start the process of shrinking the full model too early or too late. Hence, we propose DES that starts the process of shrinking the full model when it is partially trained (~50%) which leads to training cost improvement. The proposed approach also consists of novel heuristics that dynamically adjust subnet learning rates incrementally (Eps), leading to better weight-shared knowledge distillation from larger to smaller subnets. As a result, DES outperforms state-of-the-art once-for-all training techniques across different datasets including CIFAR10/100, ImageNet-100, and ImageNet-1k on accuracy and cost. It achieves 1.83% higher ImageNet-1k top1 accuracy or the same accuracy with 1.3x x reduction in FLOPs and 2.5x drop in training cost (GPU*hrs).

Degree

thesis:*
Level thesis:degree_level
Masters
Department dc:contributor.department
Computer Science
Grantor dc:publisher
Georgia Institute of Technology
Year dc:date.issued
2023

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Annavajjala, Aditya
Advisor dc:contributor.advisor
  • Tumanov, Alexey
Committee members dc:contributor.committeemember
  • Lin, Yingyan (Celine)
  • Davenport, Mark A.

Subjects

dc:subject × 4

Rights

Language dc:language.iso
en_US

Identifiers

dc:identifier.*
Handle dc:identifier.uri
https://hdl.handle.net/1853/76846

Chain of custody

source
Harvested from
Georgia Tech
Base URL
repository.gatech.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Annavajjala, Aditya. DES: Delayed Eps-Shrinking for Faster Once-For-All Training. Masters thesis, Georgia Institute of Technology, 2023. https://hdl.handle.net/1853/76846