Cut LLM Serving Costs 8x with OSCAR: A 2-Bit KV Cache Quantization Method for SGLang and vLLM
Researchers have introduced a new method to dramatically reduce the memory footprint and boost the throughput of large language model serving without significant accuracy loss, directly targeting infrastructure cost reduction.
Practical Summary
OSCAR is a new quantization technique that compresses the KV cache in LLMs to 2 bits. It uses an offline, attention-aware analysis to design rotations that align with what the model's attention mechanism actually needs. This approach avoids the accuracy collapse seen in naive quantization methods. The result is a massive 8x reduction in memory usage, up to 7x higher throughput for large batches, and 3x faster decoding, all while keeping accuracy within a few points of the full-precision BF16 baseline. Crucially, it is deployable in popular inference engines like SGLang and vLLM.
Why It Matters
For businesses running large language models at scale, the operational cost is dominated by GPU memory and compute throughput. This method offers a direct path to reduce infrastructure costs by allowing significantly more models or requests to be served on the same hardware, or to use less expensive hardware for the same workload. The 8x memory reduction and throughput gains translate to lower cloud bills and higher revenue per server, making it a critical workflow for AI-driven cost optimization.
Understanding the Problem: KV Cache Bottleneck
When serving Large Language Models (LLMs), the Key-Value (KV) cache stores the context of the conversation to avoid recomputing it for each new token. This cache grows quickly and becomes a major memory bottleneck, especially for long contexts or large batch sizes. High memory usage limits the number of requests you can handle concurrently (throughput) and forces the use of expensive, high-memory GPUs.
The OSCAR Solution: Smart 2-Bit Compression
OSCAR (Offline Spectral Covariance-Aware Rotation) addresses this by quantizing the KV cache to just 2 bits. Unlike naive methods that simply round values, OSCAR first performs an offline analysis of the model's attention patterns. It learns fixed rotations and clipping thresholds that make the data structure align better with the attention mechanism's needs, preserving critical information during aggressive compression.
