vLLM v0.22.0 adds a multi-tier KV cache offloading framework that moves cached attention states to secondary storage like local filesystems or Mooncake disk tiers, allowing operators to serve longer contexts or larger batches without exhausting GPU or CPU RAM.
Implements a pluggable multi-tier KV cache management system that automatically spills and reloads attention key-value blocks to disk or network storage when GPU/CPU memory is constrained, with explicit support for DeepSeek V4 and Python filesystem backends.
Operators running long-context or high-throughput LLM services can now handle larger batches and longer prompts without hitting out-of-memory crashes or paying for additional GPU nodes. By spilling idle KV cache blocks to local disks or distributed storage like Mooncake, the engine reduces peak RAM pressure and keeps more requests in flight, though disk I/O bandwidth and latency will become the new bottleneck to monitor. Teams should watch for increased tail latencies during cache reloads under heavy churn, verify filesystem throughput matches their concurrency targets, and test the fallback behavior when secondary storage fills up or experiences network partitions.