OSCAR: Cut LLM Inference Memory by 8x and Boost Throughput by 7x
A new 2-bit quantization method for the KV cache significantly reduces memory costs and increases serving speed for large language models with minimal accuracy loss.
Practical Summary
Researchers have developed OSCAR, a method for compressing the key-value (KV) cache in large language models to just 2 bits per value. This reduces the memory required for this critical component by 8x. By designing the compression to align with what the attention mechanism actually needs, OSCAR avoids the accuracy collapse seen in naive methods. The practical result is the ability to serve more requests with the same hardware, or to use cheaper hardware for the same workload, while keeping accuracy nearly intact.
Why It Matters
For any business or developer running or building on LLMs, the cost and speed of inference are major bottlenecks. An 8x reduction in KV-cache memory directly translates to lower GPU memory requirements and associated costs. A 7x throughput boost at large batches means serving more users or processing more data in the same time, directly impacting operational efficiency and revenue capacity. This makes deploying large models more economically viable.
What is the KV Cache and Why Does it Matter for Cost?
When a Large Language Model (LLM) generates text, it uses a key-value (KV) cache to store intermediate calculations from previous tokens. This cache grows with the sequence length and batch size, consuming significant GPU memory. Reducing its size is a direct lever for cutting inference costs, as it allows more requests to be processed concurrently on the same hardware.
The OSCAR Method: Quantizing the KV Cache to 2 Bits
OSCAR (Offline Spectral Covariance-Aware Rotation) is a technique for compressing the KV cache values to a very low 2-bit precision. The key innovation is performing an offline analysis to find optimal rotation and clipping parameters that align the compressed data with the needs of the attention mechanism. This prevents the severe accuracy loss that typically comes with extreme quantization.
Measured Business Impact: Accuracy, Memory, and Speed
The researchers tested OSCAR on models up to 358B parameters (Qwen3, GLM-4). Key results: - **Accuracy:** Maintains performance within 1–4 points of the standard BF16 baseline. - **Memory:** Reduces KV-cache memory usage by 8x. - **Throughput:** Increases serving throughput by up to 7x for large batch sizes. - **Compatibility:** The method is implemented and runnable in popular open-source inference frameworks like SGLang and vLLM.