Virginia Tech
Disaggregated LLM Serving with CXL Shared Memory KV Cache at Rack-Scale
Abstract
dc:description.abstractDisaggregated LLM serving improves resource efficiency by separating prefill and decode across workers, but introduces a new bottleneck: key/value (KV) tensors generated during prefill must be transferred to decode workers, typically through the network stack. CXL memory offers an attractive alternative by enabling multiple hosts to share a large byte-addressable memory pool, allowing KV blocks to be published and consumed without network transfer. Realizing this vision is challenging because current CXL devices provide neither cross-host atomic operations nor full-device cache coherence, and expose only raw shared memory. This thesis presents TraCT, a rack-scale LLM serving system that uses CXL shared memory as both a network-free KV-transfer substrate and a rack-scale prefix-aware KV cache. TraCT overcomes three key challenges imposed by current CXL hardware: inter-node mutual exclusion without hardware atomics, correct visibility on non-coherent shared memory, and portable shared-memory data structures without shared pointers. To this end, it introduces a two-tiered lock, a software-managed coherence discipline, and a shared object directory. Implemented as a general-purpose CXL shared-memory library with a vLLM external KV-caching plugin, TraCT achieves up to 2.6x reduction in average time-to-first-token (TTFT), 3.0x lower P99 latency, and 1.9x higher peak goodput than RDMA-based baselines on a real-world multi-turn conversational workload. These results demonstrate that CXL shared memory is a practical and efficient substrate for KV transfer and prefix caching in rack-scale disaggregated LLM serving.
Degree
thesis:*- Name thesis:degree_name
- Master of Science
- Level thesis:degree_level
- masters
- Discipline thesis:degree_discipline
- Computer Science & Applications
- Department dc:contributor.department
- Computer Science and#38; Applications
- Grantor dc:publisher
- Virginia Tech
- Year dc:date.issued
- 2026
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Yoon, Dongha
- Chair dc:contributor.committeechair
-
- Noh, Sam Hyuk
- Committee members dc:contributor.committeemember
-
- Li, Huaicheng
- Butt, Ali
Subjects
dc:subject × 4Rights
dc:rights- Statement dc:rights
-
- In Copyright
- Licence dc:rights.uri
- Language dc:language.iso
- en
Identifiers
dc:identifier.*- Dc Identifier Other
- vt_gsexam:46695