University of Minnesota
Accelerating Distributed Deep Learning via Software-Defined RDMA Transport in AI Data Centers
Abstract
dc:description.abstractArtificial intelligence models continue to grow in scale, pushing the memory and compute capacity of individual accelerators (e.g., GPUs). Training and serving trillion-parameter models therefore requires distributed execution across thousands of accelerators. In such systems, communication efficiency is essential for keeping large GPU clusters synchronized. A key property of distributed deep learning (DDL) is that computation and communication are tightly coupled: collective communication is used throughout training across parallelization strategies such as pipeline parallelism, fully sharded data parallelism, and mixture-of-experts for gradient, weight, and parameter exchange. As a result, transport-layer efficiency has a direct impact on end-to-end training throughput and cluster utilization. To meet these communication demands, AI data centers deploy Remote Direct Memory Access (RDMA) as a state-of-the-art transport mechanism. RDMA is widely used through collective communication libraries such as NCCL and Gloo, providing kernel-bypass and zero-copy communication for high-throughput, low-latency data movement. However, existing RDMA transport designs have limitations for communication-intensive AI training in shared AI data center networks (DCNs). RDMA’s single-path transport can underutilize the path diversity of modern spine–leaf topologies, leaving large collective flows on individual links. In addition, RDMA’s single-queue design, combined with lossless mechanisms such as Priority Flow Control (PFC), can introduce queue-level blocking and performance interference under the bursty, synchronized traffic patterns of DDL workloads. This dissertation shows that software-defined and virtualized RDMA transport architectures, supporting multi-path and multi-queue communication, can improve DDL performance while improving network efficiency. Rather than relying on specialized hardware, we show that end-host software mechanisms can decouple RDMA transport control from fixed hardware behavior and enable deployable transport optimizations. We first design and implement Maestro, a software-based multi-path RDMA framework that operates in user space and enables fine-grained path control and dynamic load balancing across multiple network paths without sacrificing RDMA’s zero-copy semantics. Maestro implements a virtualized NIC abstraction that allows applications to dynamically steer flows across network paths, enabling intelligent path selection and congestion avoidance at the end host. We evaluate Maestro on a real-world testbed, demonstrating up to 2× improvement in collective communication performance by effectively utilizing multiple paths to avoid congestion. We then present MQ-RDMA, a multi-queue RDMA transport framework tailored for lossless data center networks that combines proactive application-layer prediction with reactive congestion notifications. By leveraging multiple priority queues with dynamic queue scheduling, MQ-RDMA mitigates PFC-induced blocking and improves collective communication throughput. Experiments on a real-world testbed and large-scale simulation show that MQ-RDMA achieves up to 8.3× improvement in victim-flow throughput and 40% reduction in collective completion time compared to state-of-the-art congestion control. Together, Maestro and MQ-RDMA show that moving RDMA transport control into end-host software provides a flexible way to use application-level information, adopt new scheduling and load-balancing mechanisms, and reduce communication bottlenecks in distributed deep learning workloads without hardware modifications.
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Tian, Feng
Rights
- Language dc:language.iso
- en
Identifiers
dc:identifier.*- Dc Identifier Citation
- Tian, Feng. (2026). Accelerating Distributed Deep Learning via Software-Defined RDMA Transport in AI Data Centers. Retrieved from the University Digital Conservancy, https://hdl.handle.net/11299/283160.