Optimize LLM Inference Throughput: vLLM's Triton Fast-Path for KV Offloading
A targeted kernel optimization that bypasses a DMA performance cliff for small memory copies, doubling requests per second and cutting latency in offloaded LLM serving.
Practical Summary
A change to vLLM's offloading connector replaces the standard CUDA DMA path for small, frequent CPU-to-GPU memory transfers with a custom Triton kernel. This addresses a specific performance bottleneck where small copies are extremely slow, directly increasing the throughput and reducing the cost of serving large language models that use memory offloading.
Why It Matters
For teams running large language models with offloading to manage GPU memory costs, this optimization offers a concrete way to significantly improve serving capacity and reduce latency without hardware changes. Faster inference directly translates to lower cost-per-query and higher potential revenue from the same infrastructure.
Understanding the Performance Bottleneck: The DMA Small-Page Cliff
The standard `cuMemcpyBatchAsync` function used for copying KV cache data between CPU and GPU memory is highly efficient for large, contiguous blocks. However, for the small, scattered memory descriptors (typically 4-24 KiB) common in KV offloading, its performance collapses. This creates a 'cliff' where throughput stagnates around 5-7 GB/s, severely limiting the overall speed of memory transfers critical for LLM inference.
The Solution: A Triton Kernel for Small Copies
The optimization introduces a small Triton kernel (`_swap_blocks_kernel`) that takes over the CPU-to-GPU copy direction for payloads smaller than 28 KiB. This kernel is more efficient at handling the many small transfers required during KV offloading. The choice of 28 KiB as the threshold is based on benchmarking data showing this is the crossover point where the new kernel's performance surpasses the DMA path.