Back to search

Georgia Institute of Technology

Methodologies for co-designing supercomputer-scale systems and deep learning software

Abstract

dc:description.abstract

This dissertation introduces new methodologies to co-design deep learning software and supercomputer hardware in the setting of large-scale training. The first is an analytical performance model for exploring the co-design space of parallel algorithms for large language models (LLMs) and potential supercomputer architectures during the early phases of the co-design process. On the algorithm side, we consider diverse implementation strategies, including data, tensor, and pipeline parallelism, communication-computation overlap, and memory optimization. The hardware aspect includes hierarchical memory systems, multiple interconnection networks, and parameterized efficiencies based on operation size. Our open-source tool, Calculon, implements this model. Its analytical nature enables rapid evaluation, estimating performance for billions of strategy and architecture combinations. This facilitates co-design-space exploration for future LLMs with trillions of parameters, yielding insights into optimal system characteristics and the interplay between algorithmic and architectural decisions. As models scale beyond 100 trillion parameters, two bottlenecks become especially critical to address: memory capacity and network speed. For the former, Calculon suggests a hardware solution involving the addition of slower capacity-tier memory for intermediate tensors and model parameters, optimizing faster memory for current layer computation. For the latter, we present novel distributed-memory parallel matrix multiplication algorithms capable of hiding communication entirely, potentially achieving perfect scaling. Looking ahead, we foresee a need to model artificial intelligence (AI) applications beyond LLMs and perform detailed system simulations in later design stages. To meet these demands, we introduce ParaGraph, a tool bridging the gap between applications and network hardware simulators. ParaGraph features a high-level graph representation of parallel programs, automatically extracted from compiled applications and a runtime environment for emulator-based dynamic execution. Case studies on deep learning workloads extracted from JAX and TensorFlow programs illustrate ParaGraph's utility for software-hardware co-design workflows, including communication optimization, hardware bottleneck identification, and simulation validation.

Degree

thesis:*
Level thesis:degree_level
Doctoral
Department dc:contributor.department
Computational Science and Engineering
Grantor dc:publisher
Georgia Institute of Technology
Year dc:date.issued
2024

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Isaev, Mikhail
Advisor dc:contributor.advisor
  • Vuduc, Richard
Committee members dc:contributor.committeemember
  • Young, Jeffrey
  • McDonald, Nicholas G.
  • Krishna, Tushar
  • Kim, Hyesoon

Subjects

dc:subject × 11

Rights

Language dc:language.iso
en_US

Identifiers

dc:identifier.*
Handle dc:identifier.uri
https://hdl.handle.net/1853/75228

Chain of custody

source
Harvested from
Georgia Tech
Base URL
repository.gatech.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Isaev, Mikhail. Methodologies for co-designing supercomputer-scale systems and deep learning software. Doctoral thesis, Georgia Institute of Technology, 2024. https://hdl.handle.net/1853/75228