Georgia Institute of Technology
Leveraging sparsity in deep neural networks for training efficiency, interpretability and generalization
Abstract
dc:description.abstractSparse neural networks (Sparse NNs) are characterized by having fewer connections between consecutive layers compared to traditional fully connected, or dense NNs. Historically, sparsity has been studied post-training to enhance inference efficiency and as a regularization mechanism to improve generalization. However, additional benefits beyond these areas remain underexplored. In this thesis, we investigate sparse NNs, various sparsity patterns, and their broader benefits, including improved training efficiency, enhanced interpretability, and stronger generalization. First, we introduce PHEW (Path with Higher Edge-Weights), a novel method for identifying sparse sub-networks within dense NNs at initialization, without using any training data. PHEW is a probabilistic network formation method based on biased random walks, relying solely on the initial weights of the NN. Importantly, PHEW does not make any task-specific assumptions; instead, it exploits structural properties inherent in dense NNs that promote faster convergence and better generalization. By identifying effective sparse sub-networks at initialization, PHEW reduces the computational burden of training dense NNs and consistently outperforms other state-of-the-art methods. Second, we propose Neural Sculpting, a technique to uncover the underlying hierarchically modular task structure within NNs. Many real-world tasks exhibit hierarchical modularity, where complex target functions can be decomposed into simpler sub-functions arranged in a hierarchy. We pose the following question: given a sufficiently deep NN, how can we uncover the task’s hierarchical structure? Neural Sculpting uses an iterative process of pruning both units and edges during training, followed by network analysis to detect functional modules and infer hierarchical relationships between them. This method enhances the interpretability of NNs by guiding them to reflect the task’s inherent hierarchical and modular structure through pruning, and subsequently revealing that structure through network analysis. Finally, we leverage structural information about the task’s hierarchical modularity to enhance NN performance by aligning the architecture at initialization with the task’s structure. Specifically, we investigate how modular NNs can outperform dense NNs by systematically varying the degree of structural knowledge incorporated at initialization. We compare architectures ranging from monolithic dense NNs, which assume no prior knowledge, to hierarchically modular NNs with shared modules, which leverage sparsity, modularity, and module reusability. Incorporating modularity and module reuse significantly enhances learning efficiency and generalization, particularly in data-scarce scenarios, where hierarchically modular NNs excel by promoting functional specialization and reducing redundancy. These findings reveal that task-specific architectural biases can lead to more efficient, interpretable, and effective learning systems. In conclusion, this thesis demonstrates that sparse NNs offer not only enhanced training and inference efficiency but also superior interpretability and generalization capabilities. These findings have broad implications for NN design across various domains, particularly in data-scarce scenarios or applications where understanding the underlying task structure is essential. Future work may focus on refining these methodologies and extending their applicability to more complex, real-world tasks and larger-scale architectures.
Degree
thesis:*- Level thesis:degree_level
- Doctoral
- Department dc:contributor.department
- Electrical and Computer Engineering
- Grantor dc:publisher
- Georgia Institute of Technology
- Year dc:date.issued
- 2024
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Malakarjun Patil, Shreyas
- Advisor dc:contributor.advisor
-
- Dovrolis, Constantine
- Committee members dc:contributor.committeemember
-
- Liu, Ling
- Kira, Zsolt
- Lin, Yingyan (Celine)
- Michael, Loizos
Subjects
dc:subject × 3Rights
- Language dc:language.iso
- en_US
Identifiers
dc:identifier.*- Handle dc:identifier.uri
- https://hdl.handle.net/1853/76897