vLLM now includes a native HIP W4A16 linear kernel optimized for AMD RDNA3 GPUs, replacing the slow Triton fallback and the fp16-only ExLlama path with a high-throughput implementation that natively supports bf16 and fp16.
Implements a device-specific HIP GEMM kernel that uses scalar dot-products for decode and WMMA instructions for prefill, eliminating the need to downcast bf16 models to fp16 and bypassing Triton's overhead on consumer AMD GPUs.
Operators running quantized LLMs on consumer AMD Radeon GPUs can now serve bf16 models at production speeds without encountering numerical instability or falling back to unusably slow fallbacks. By providing a native HIP kernel that handles bf16 natively at 2.5–4.2× the throughput of the previous Triton path, this change removes the trade-off between speed and output quality for RDNA3 deployments. Users no longer need to manually cast weights to fp16, which previously caused inf/NaN propagation and garbled tool-calling outputs under load. Teams should monitor register pressure and WMMA dispatch behavior on varying batch sizes, as the kernel currently restricts WMMA acceleration to bf16 and relies on scalar paths for fp16 decode.