Back to search

Georgia Institute of Technology

Efficient Serving for Large Language Model-based Multi-agent System

Abstract

dc:description.abstract

This thesis presents a comprehensive study on improving the serving efficiency of large language model (LLM) based multi-agent systems through a new mixture-of-agents architecture called Faster-MoA. Traditional mixture-of-agents (MoA) systems connect agents in a fully connected manner, which causes significant communication overhead and synchronization stalls that slow down inference. To tackle these challenges, this work proposes an integrated algorithm-system co-design that simultaneously reduces redundant agent interactions, dynamically prunes unnecessary computation and overlaps prefilling and decoding stages. The thesis first reviews the foundations of mixture-of-experts (MoE) and MoA architectures. MoE divides a model into specialist sub-networks and employs a gating network to route inputs to the most suitable experts. While MoE provides efficiency gains in training and inference, its translation to multiple independent LLM agents creates unique challenges in agent communication and scheduling. Prior MoA systems adopt dense all-to-all connectivity among agents, which limits scalability and wastes computation. To address these inefficiencies, the thesis introduces three key innovations: • Hierarchical tree-structured agent topology. Agents are grouped into clusters, and each agent in the next layer aggregates information only from its assigned cluster rather than from all agents. This local aggregation followed by global aggregation reduces redundant inter-agent communication while preserving essential context. • Semantic-guided dynamic early exit (EE). During inference, an adaptive mechanism evaluates the semantic similarity and confidence of intermediate outputs across agents. If smaller agents already produce sufficiently high-quality answers, connections to larger agents are pruned at run time, avoiding expensive computations without degrading answer quality. • Agent-dependency-aware incremental prefilling. Prefill and decode stages are overlapped by streaming decoded tokens from precursor agents to successor agents through an agent prefill cache. This incremental prefilling reduces idle GPU time and increases hardware utilization. Experiments on five benchmarks: GSM8K, MATH-500, AIME2025, MMLU-ProX- Lite and IFBench, demonstrate that Faster-MoA significantly reduces end-to-end latency compared with traditional all-to-all MoA systems. For example, integrating the tree structure alone reduces latency by up to 62 %, while combining dynamic early exit and incremental prefilling yields up to 90 % latency reduction without sacrificing accuracy. In some cases, dynamic early exit even improves accuracy by suppressing noisy outputs from weaker agents. Hardware experiments on NVIDIA H200 GPUs show that the proposed incremental prefilling provides superior overlap between prefilling and decoding stages compared with data-parallel and chunked prefill baselines. Overall, this thesis delivers a principled and practical framework for efficient LLM-based MoA serving. The proposed hierarchical tree topology, dynamic early exit and incremental prefilling are complementary innovations that enable scalable, low-latency MoA systems. The findings highlight the importance of co-designing algorithms and systems to unlock the full potential of next-generation intelligent agents

Degree

thesis:*
Name thesis:degree_name
Electrical and Computer Engineering, MS
Department thesis:degree_department
Electrical and Computer Engineering
Grantor
Georgia Institute of Technology
Year dc:date.issued
2026

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Wang, Zijun
Advisor dc:contributor.advisor
  • Hao, Cong (Callie)
Committee members dc:contributor.committeemember
  • Krishna, Tushar
  • Mahajan, Divya

Identifiers

dc:identifier.*
Handle dc:identifier.uri
https://hdl.handle.net/1853/81624

Chain of custody

source
Harvested from
Georgia Tech
Base URL
repository.gatech.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
related terms
citation

Wang, Zijun. Efficient Serving for Large Language Model-based Multi-agent System. Georgia Institute of Technology, 2026. https://hdl.handle.net/1853/81624