Georgia Institute of Technology
Methodologies for co-designing supercomputer-scale systems and deep learning software
Abstract
dc:description.abstractThis dissertation introduces new methodologies to co-design deep learning software and supercomputer hardware in the setting of large-scale training. The first is an analytical performance model for exploring the co-design space of parallel algorithms for large language models (LLMs) and potential supercomputer architectures during the early phases of the co-design process. On the algorithm side, we consider diverse implementation strategies, including data, tensor, and pipeline parallelism, communication-computation overlap, and memory optimization. The hardware aspect includes hierarchical memory systems, multiple interconnection networks, and parameterized efficiencies based on operation size. Our open-source tool, Calculon, implements this model. Its analytical nature enables rapid evaluation, estimating performance for billions of strategy and architecture combinations. This facilitates co-design-space exploration for future LLMs with trillions of parameters, yielding insights into optimal system characteristics and the interplay between algorithmic and architectural decisions. As models scale beyond 100 trillion parameters, two bottlenecks become especially critical to address: memory capacity and network speed. For the former, Calculon suggests a hardware solution involving the addition of slower capacity-tier memory for intermediate tensors and model parameters, optimizing faster memory for current layer computation. For the latter, we present novel distributed-memory parallel matrix multiplication algorithms capable of hiding communication entirely, potentially achieving perfect scaling. Looking ahead, we foresee a need to model artificial intelligence (AI) applications beyond LLMs and perform detailed system simulations in later design stages. To meet these demands, we introduce ParaGraph, a tool bridging the gap between applications and network hardware simulators. ParaGraph features a high-level graph representation of parallel programs, automatically extracted from compiled applications and a runtime environment for emulator-based dynamic execution. Case studies on deep learning workloads extracted from JAX and TensorFlow programs illustrate ParaGraph's utility for software-hardware co-design workflows, including communication optimization, hardware bottleneck identification, and simulation validation.
Degree
thesis:*- Level thesis:degree_level
- Doctoral
- Department dc:contributor.department
- Computational Science and Engineering
- Grantor dc:publisher
- Georgia Institute of Technology
- Year dc:date.issued
- 2024
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Isaev, Mikhail
- Advisor dc:contributor.advisor
-
- Vuduc, Richard
- Committee members dc:contributor.committeemember
-
- Young, Jeffrey
- McDonald, Nicholas G.
- Krishna, Tushar
- Kim, Hyesoon
Subjects
dc:subject × 11Rights
- Language dc:language.iso
- en_US
Identifiers
dc:identifier.*- Handle dc:identifier.uri
- https://hdl.handle.net/1853/75228