Georgia Institute of Technology
Efficient Serving for Large Language Model-based Multi-agent System
Abstract
dc:description.abstractThis thesis presents a comprehensive study on improving the serving efficiency of large language model (LLM) based multi-agent systems through a new mixture-of-agents architecture called Faster-MoA. Traditional mixture-of-agents (MoA) systems connect agents in a fully connected manner, which causes significant communication overhead and synchronization stalls that slow down inference. To tackle these challenges, this work proposes an integrated algorithm-system co-design that simultaneously reduces redundant agent interactions, dynamically prunes unnecessary computation and overlaps prefilling and decoding stages. The thesis first reviews the foundations of mixture-of-experts (MoE) and MoA architectures. MoE divides a model into specialist sub-networks and employs a gating network to route inputs to the most suitable experts. While MoE provides efficiency gains in training and inference, its translation to multiple independent LLM agents creates unique challenges in agent communication and scheduling. Prior MoA systems adopt dense all-to-all connectivity among agents, which limits scalability and wastes computation. To address these inefficiencies, the thesis introduces three key innovations: • Hierarchical tree-structured agent topology. Agents are grouped into clusters, and each agent in the next layer aggregates information only from its assigned cluster rather than from all agents. This local aggregation followed by global aggregation reduces redundant inter-agent communication while preserving essential context. • Semantic-guided dynamic early exit (EE). During inference, an adaptive mechanism evaluates the semantic similarity and confidence of intermediate outputs across agents. If smaller agents already produce sufficiently high-quality answers, connections to larger agents are pruned at run time, avoiding expensive computations without degrading answer quality. • Agent-dependency-aware incremental prefilling. Prefill and decode stages are overlapped by streaming decoded tokens from precursor agents to successor agents through an agent prefill cache. This incremental prefilling reduces idle GPU time and increases hardware utilization. Experiments on five benchmarks: GSM8K, MATH-500, AIME2025, MMLU-ProX- Lite and IFBench, demonstrate that Faster-MoA significantly reduces end-to-end latency compared with traditional all-to-all MoA systems. For example, integrating the tree structure alone reduces latency by up to 62 %, while combining dynamic early exit and incremental prefilling yields up to 90 % latency reduction without sacrificing accuracy. In some cases, dynamic early exit even improves accuracy by suppressing noisy outputs from weaker agents. Hardware experiments on NVIDIA H200 GPUs show that the proposed incremental prefilling provides superior overlap between prefilling and decoding stages compared with data-parallel and chunked prefill baselines. Overall, this thesis delivers a principled and practical framework for efficient LLM-based MoA serving. The proposed hierarchical tree topology, dynamic early exit and incremental prefilling are complementary innovations that enable scalable, low-latency MoA systems. The findings highlight the importance of co-designing algorithms and systems to unlock the full potential of next-generation intelligent agents
Degree
thesis:*- Name thesis:degree_name
- Electrical and Computer Engineering, MS
- Department thesis:degree_department
- Electrical and Computer Engineering
- Grantor
- Georgia Institute of Technology
- Year dc:date.issued
- 2026
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Wang, Zijun
- Advisor dc:contributor.advisor
-
- Hao, Cong (Callie)
- Committee members dc:contributor.committeemember
-
- Krishna, Tushar
- Mahajan, Divya
Identifiers
dc:identifier.*- Handle dc:identifier.uri
- https://hdl.handle.net/1853/81624