OSCAR: Cut LLM Inference Costs 8x with 2-bit KV Cache Quantization
A new method from Together AI, University of Sydney, and UIUC uses offline attention-aware analysis to achieve near-BF16 accuracy with 2-bit KV cache, slashing memory use and boosting throughput for large-batch AI workloads.
Practical Summary
OSCAR is a technique that applies fixed rotations and clipping thresholds to compress the key-value (KV) cache in large language models to 2-bit precision. By aligning the compression with what the attention mechanism actually needs, it avoids the accuracy collapse seen in naive methods. The result is an 8x reduction in KV-cache memory, up to 7x higher throughput for large batches, and 3x faster decoding than BF16, all while maintaining accuracy within 1-4 points of the original model. It is deployable in popular serving frameworks like SGLang and vLLM.
Why It Matters
For businesses running large language models, inference costs—especially memory and compute for long contexts—are a major operational expense. OSCAR offers a practical way to significantly reduce these costs (8x memory savings) and increase serving capacity (7x throughput at scale) without a major accuracy trade-off. This directly translates to lower cloud bills, the ability to serve more users with the same hardware, and improved profit margins on AI-powered services, which is core to revenue growth through efficiency.
Understanding the KV Cache Bottleneck
In Transformer-based LLMs, the key-value (KV) cache stores intermediate attention computations for each token in a sequence. As sequence length grows, this cache consumes enormous amounts of GPU memory, limiting batch sizes and increasing costs. Standard 16-bit (BF16) precision is accurate but memory-intensive. Naively quantizing to 2-bit (INT2) causes catastrophic accuracy loss.
How OSCAR Works: Aligning Compression with Attention
OSCAR stands for Offline Spectral Covariance-Aware Rotation. Instead of compressing values blindly, it performs an offline analysis of the covariance structure of the KV cache to design optimal rotations. These rotations transform the data into a form where aggressive 2-bit quantization causes minimal harm. Fixed clipping thresholds are then applied. The key insight is aligning the compressed representation with what the attention mechanism subsequently needs.
Because the rotation and clipping parameters are computed offline, the runtime overhead is minimal, making it efficient for deployment.
