4.4x – 24x Throughput Gains: How the Updated Ray Serve LLM + vLLM Stack Slashes Inference Costs
A new integration between Ray Serve LLM and vLLM delivers major throughput improvements for both prefill and decode workloads, offering a concrete path to lower serving costs or higher revenue per GPU.
Practical Summary
This update from the Ray and vLLM teams, in partnership with Google Cloud, shows a production-ready stack that significantly boosts LLM inference throughput. The gains—up to 4.4x for prefill-heavy tasks and 24x for decode-heavy tasks—come from three targeted optimizations to routing and execution. For teams running AI-powered products, this translates directly into lower cost-per-token or higher service capacity on existing hardware.
Why It Matters
For businesses scaling AI, inference cost and latency are critical bottlenecks. This benchmark demonstrates a concrete, open-source-based workflow to achieve order-of-magnitude throughput improvements without changing models. This can reduce cloud spending, enable serving more users on the same budget, or allow offering lower-latency premium tiers—all of which directly impact revenue growth and unit economics.
Understanding the Performance Breakthrough
The core update is that Ray Serve LLM, a serving framework, now integrates deeply with the vLLM engine and achieves throughput competitive with specialized, high-performance routers. The measured improvements are dramatic: up to 4.4x for prefill-heavy workloads (like processing long prompts) and up to 24x for decode-heavy workloads (like generating long responses). This performance leap comes from three specific optimizations to the serving stack's architecture.
The Three Key Optimizations Explained
1. **Direct Streaming with Control-Plane-Only Endpoint Picker:** This optimization streamlines how requests are routed to available model instances, reducing coordination overhead. It allows for more direct data flow, which is critical for high-throughput scenarios.
2. **A New vLLM Ray V2 Executor Backend:** This is a new backend for the vLLM engine when running on Ray. It likely improves how inference tasks are scheduled and executed across a cluster of machines, maximizing hardware utilization.
3. **HAProxy Ingress for Routing at the Speed of C:** By using HAProxy, a high-performance, C-based load balancer, for request routing at the network edge, the stack eliminates a potential software bottleneck. This ensures incoming traffic is handled with minimal latency before it reaches the AI inference layer.