Achieve 1000+ tokens/sec Prefill on an 8GB GPU with Gemma 4 12B and TurboQuant
A specific llama.cpp configuration trick dramatically boosts inference speed and context length on consumer hardware, lowering the cost of running capable AI models.
Practical Summary
Using a fork of llama.cpp with TurboQuant KV cache optimizations, the Gemma 4 12B dense model can prefill at over 1000 tokens per second on a single 8GB GPU, while supporting a 120k token context window. This represents a major hardware cost reduction for teams running local inference.
Why It Matters
For businesses building AI products or automations, this technique directly reduces the required hardware investment for high-throughput inference. Running a 12B parameter model with vision and audio capabilities on a common 8GB graphics card opens new, cost-effective deployment scenarios for internal tools and customer-facing features.
The Performance Gains from TurboQuant Cache Optimization
This method uses a specific fork of llama.cpp called TurboQuant, developed by Tom Turney. The key is two KV cache flags that dramatically reduce memory usage, allowing a larger model and longer context to fit into 8GB of VRAM. Compared to a baseline run without these flags, users report a 42% increase in prefill speed, a 25% increase in decode speed, and a 150% increase in usable context length.
The core innovation is not a new model quantization but a memory-efficient representation of the Key-Value cache during inference. This is what 'frees up the room' to run the full 12B dense model on a consumer RTX 4060 8GB card.
Step-by-Step Setup Command
To replicate this setup, you need the TurboQuant fork of llama.cpp and the corresponding quantized model file. The exact command line used for testing is below. It specifies the model, a 120k context window, and the critical cache type flags.
Command: llama.cpp TurboQuant flags: -m gemma-4-12B-it-qat-UD-Q4_K_XL.gguf -c 120000 --cache-type-k q8_0 --cache-type-v turbo3 -ngl 99 --port 8080
Key parameters: `-c 120000` sets the context window to 120,000 tokens. `-ngl 99` offloads all model layers to the GPU. The `--cache-type-k q8_0 --cache-type-v turbo3` flags implement the TurboQuant KV cache optimization.
