Global ETD Search
Search theses and dissertations gathered from participating repositories worldwide. Every result links back to the library that holds it. No account is needed.
Results
Showing 1 to 20 of 29 for “"KV Cache"”.
-
Dynamic KV cache management for efficient LLM inference
… memory-intensive. To accelerate generation, LLMs cache previously computed information in a Key-Value (KV) cache. As task complexity and sequence lengths grow, the KV cache becomes the primary memory bottleneck for LLM inference, limiting their ability to scale. In this thesis, we study the …
-
End-to-End KV-Cache Materialization and Evaluation
Key/value (KV) caching is essential for efficient Transformer inference, but its size grows with context length and model depth, creating bottlenecks in memory capacity and data movement. In distributed and multi-tier serving, these overheads directly affect Time-to-First-Token (TTFT), making …
-
Disaggregated LLM Serving with CXL Shared Memory KV Cache at Rack-Scale
… but introduces a new bottleneck: key/value (KV) tensors generated during prefill must be transferred to decode workers, typically through the network stack. CXL memory offers an attractive alternative by enabling multiple hosts to share a large byte-addressable memory pool, allowing KV blocks …
-
Pushing the limits of long context LLM inference via KV cache compression
Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2025-10-19 without embargo terms
-
Ghostdecoding: leveraging random-feature kernels for error-aware and training-free KV cache selection
Submission published under a 24 month embargo labeled 'Closed Access', the embargo will last until 2027-05-01
-
Low-rank KV cache compression in LLMs: A second-order output metric and its limits
The key–value (KV) cache makes autoregressive decoding in large language models fast, but it grows with the length of the context. At long context it becomes the main drain on a GPU’s memory, both in how much it has to store and in how fast it can be read. Compressing the keys and values with a …
-
Adaptive KV Cache Management for Efficient Transformer-based LLM Inference - Leveraging Attention Sparsity for Memory Optimization
… prohibitive memory footprint of the Key-Value (KV) cache. As LLMs scale, the KV cache becomes a significant bottleneck, limiting longer context windows and overall operational efficiency. To mitigate this issue, we propose and evaluate Adap-KV, a novel adaptive memory management strategy for the …
-
Efficient Long-Context Retrieval Systems via Dynamic Sparse Attention
… corpus scale and context length, the key-value (KV) cache makes serving them impractical due to its linear growth with sequence length and batch size. Serving systems like RetroInfer use dynamic sparse attention (DSA) to address that bottleneck, offloading the cache and attending at each decode …
-
Quantized Lookup Table-Based Representation for Key-Value Cache
… bottleneck: the transmission of Key-Value (KV) caches between devices to enable cache reuse. As the demand for long-context inference grows, KV caches can become extremely large, making efficient compression essential to reduce communication costs. Interestingly, we observe that KV caches …
-
Quantized Lookup Table-Based Representation for Key-Value Cache
… bottleneck: the transmission of Key-Value (KV) caches between devices to enable cache reuse. As the demand for long-context inference grows, KV caches can become extremely large, making efficient compression essential to reduce communication costs. Interestingly, we observe that KV caches …
-
Quantized Lookup Table-Based Representation for Key-Value Cache
… bottleneck: the transmission of Key-Value (KV) caches between devices to enable cache reuse. As the demand for long-context inference grows, KV caches can become extremely large, making efficient compression essential to reduce communication costs. Interestingly, we observe that KV caches …
-
The Lookahead Transformer
… stage processes all input tokens in parallel and caches them into the KV cache. The decode stage uses the KV cache, processes a single token at a time, and is generally memory-bandwidth constrained. Each decode step generates a token probability distribution, samples from it, appends the token to …
-
Memory-Efficient and Scalable Parallel Approaches for High-Throughput Machine Learning Systems
… long-sequence decoding, where Key-Value (KV) cache access can account for over 80% of per-token latency. Existing parallelism strategies, including data, tensor, and pipeline parallelism, either expose limited intra-layer parallelism or rely on redundant replication of model weights and KV …
-
Memory-Efficient and Scalable Parallel Approaches for High-Throughput Machine Learning Systems
… long-sequence decoding, where Key-Value (KV) cache access can account for over 80% of per-token latency. Existing parallelism strategies, including data, tensor, and pipeline parallelism, either expose limited intra-layer parallelism or rely on redundant replication of model weights and KV …
-
Extracting Hidden, Protected, and Deleted User Interaction Contexts from Applications and AI Systems
… by extracting and realigning batch tokens and KV cache state so investigators can interrogate a live session with recovered context.
-
EFFECTIVE COMPRESSION FOR SCALABLE VISION GENERATIVE MODELS
… tokenizer capacity, context length and cache size, and diffusion sampling steps. First, SeQ-GAN introduces semantic compression, prioritizing high-level structure and texture over pixel-perfect reconstruction to create a latent space that generative transformers can model more …
-
Energy-Efficient Hardware Acceleration of Transformer-Based Models
… activations, the proposed Weight-Hessian-aware KV cache quantization applies intra-layer mixed-precision using precomputed sensitivities, eliminating runtime overhead. To further improve hardware efficiency, the proposed Query-Key coupled scheme aligns bit precision within each outer product …
-
Compression Algorithms for Efficient Inference and Adaptation of Foundation Models
… a method for compressing the key/value (KV) cache in transformer models. We observe that KV cache channels exhibit significant redundancy, which existing quantization methods fail to exploit. Coupled Quantization jointly compresses multiple channels by leveraging their interdependence, …
-
Linear recurrent models for robust and interpretable long-context modeling
… of Transformers during training and linear KV-cache growth at inference makes them computationally prohibitive for such tasks, necessitating the exploration of more efficient architectures. Linear-time recurrent models and hybrid architectures, which combine linear recurrence with local …
-
On the Resource Efficiency of Language Models
… distributions. LoRC progressively compresses KV cache, substantially reducing GPU memory requirements while preserving performance. These complementary thrusts form a framework for efficient, deployable language models. The findings demonstrate that targeted resource optimization enables …
Page 1 of 2