Back to search

Georgia Institute of Technology

Network-aware device placement search for distributed training

Abstract

dc:description.abstract

As deep learning models grow in scale and complexity, efficient distributed training requires not only advanced parallelization strategies but also intelligent placement of model components across heterogeneous computing infrastructures. Existing device placement frameworks often assume simplified, uniform network topologies, leading to suboptimal performance in real-world data centers where communication costs vary significantly across nodes. I present my thesis on Network-aware, efficient device placement framework based on structured dynamic programming techniques (NEST). NEST jointly optimizes device placement and parallelism configuration by explicitly modeling the hierarchical and over-subscribed nature of modern data center networks. It supports a broad range of parallelization strategies–including tensor, pipeline, data, expert, and Zero Redundancy Optimizer (ZeRO) parallelism—and integrates detailed memory and communication cost modeling. Through structured dynamic programming, NEST explores the vast placement space efficiently and offers provable optimality guarantees within its search scope. Evaluations across realistic workloads and network settings show that NEST consistently outperforms manual and network-unaware baselines, delivering significant improvements in training throughput and resource utilization.

Degree

thesis:*
Level thesis:degree_level
Masters
Department dc:contributor.department
Electrical and Computer Engineering
Grantor dc:publisher
Georgia Institute of Technology
Year dc:date.issued
2025

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Venkata, Vishnu Varma
Advisor dc:contributor.advisor
  • Mahajan, Divya
Committee members dc:contributor.committeemember
  • Hao, Cong
  • Krishna, Tushar

Subjects

dc:subject × 8

Rights

Language dc:language.iso
en_US

Identifiers

dc:identifier.*
Handle dc:identifier.uri
https://hdl.handle.net/1853/77921

Chain of custody

source
Harvested from
Georgia Tech
Base URL
repository.gatech.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Venkata, Vishnu Varma. Network-aware device placement search for distributed training. Masters thesis, Georgia Institute of Technology, 2025. https://hdl.handle.net/1853/77921