Back to search

University of Washington

Towards More Efficient Communication for Distributed Learning Systems

Abstract

dc:description.abstract

The explosion of data volume and ever-increasing speed of accelerators shift the bottleneckof large-scale distributed training tasks from computation to communication. We observesignificant pressure on the communication backends of various mainstream learning systemsin multiple environments when running such tasks. Achieving efficient large scale learningrelies on more effective communication planes. We provide detailed analysis that root-causes the bottlenecks affecting the communica-tion efficiency of these systems in the context of different environments. We pinpoint suchbottlenecks from the software, hardware and network infrastructure stacks. We show how these obstacles can be overcome with a systematic codesign of a streamlinedcommunication stack, a balanced hardware and cluster configuration with the distributedtraining workload, together with awareness of network topology and environment. We showthis series of approaches, named Parameter Box, Parameter Hub, Parameter Link along with Cloud Collectives, accelerate distributed training from small clusters to datacenters and allthe way to the commercial clouds while providing varying degrees of customization to suitdifferent needs.

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Luo, Liang
Advisors dc:contributor.advisor
  • Ceze, Luis
  • Krishnamurthy, Arvind

Subjects

dc:subject × 2

Rights

dc:rights
Statement dc:rights
  • CC BY
Language dc:language.iso
en_US

Identifiers

dc:identifier.*
Handle dc:identifier.uri
http://hdl.handle.net/1773/46428

Chain of custody

source
Harvested from
University of Washington
Base URL
digital.lib.washington.edu/server/oai/request
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Luo, Liang. Towards More Efficient Communication for Distributed Learning Systems. 2020. http://hdl.handle.net/1773/46428