Georgia Institute of Technology
The Hidden Structure of Deep Models: Discovering and Exploiting Sparse Subnetworks
Abstract
dc:description.abstractDeep learning's growing scale makes computational, memory, and communication efficiency increasingly important, and sparsity offers a principled way to improve all three. This dissertation treats sparsity as an integral component of learning in three settings: controllable sparse projection, sparse reinforcement learning, and communication-efficient federated learning. First, Group Sparse Projection (GSP) directly controls the average Hoyer sparsity of grouped vectors through one interpretable parameter while allowing individual sparsity levels to adapt. The dissertation establishes the projection's theoretical properties and algorithm, then demonstrates it in sparse nonnegative matrix factorization and neural network pruning. Second, Data Adaptive Pathway Discovery (DAPD) learns sparse reinforcement-learning pathways under changing data distributions. It adapts masks during a warm-up phase and freezes them for stable optimization. Across single-task, multi-task, online, and offline settings, DAPD preserves strong policy performance with a small fraction of a shared network's parameters. Third, Salient Sparse Federated Learning (SSFL) addresses communication-constrained, non-IID federated learning by aggregating clients' private-data saliency scores into a shared mask before training. Experiments across CIFAR-10, CIFAR-100, Tiny-ImageNet, and deeper ResNets show competitive accuracy with less communication. A geographically distributed deployment yields wall-clock savings, and one-shot mask rediscovery adapts to out-of-distribution classes. Together, these methods show that matching sparsity discovery to the learning problem can reduce parameter, computation, and communication costs while preserving competitive performance.
Degree
thesis:*- Name thesis:degree_name
- Electrical and Computer Engineering, PhD
- Department thesis:degree_department
- Electrical and Computer Engineering
- Grantor
- Georgia Institute of Technology
- Year dc:date.issued
- 2026
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Ohib, Riyasat
- Advisor dc:contributor.advisor
-
- Calhoun, Vince D.
- Committee members dc:contributor.committeemember
-
- Plis, Sergey
- Anderson, David
- Xiong, Li
- Malek, Mani
- Essa, Irfan
- ORCID person:identifier.orcid
Rights
- Language dc:language.iso
- English
Identifiers
dc:identifier.*- Handle dc:identifier.uri
- https://hdl.handle.net/1853/82225