Run Gemma 4 12B at 1000+ tok/s Prefill on an 8GB GPU with 120k Context
A specific llama.cpp workflow using TurboQuant KV cache optimization to drastically boost performance and context size for a dense 12B model on consumer hardware.
Practical Summary
A user demonstrates that Google's Gemma 4 12B dense model, when quantized (QAT) and run through a TurboQuant-modified version of llama.cpp, can achieve prefill speeds over 1000 tokens per second on a single RTX 4060 (8GB VRAM) while maintaining a 120k token context window. This represents a significant improvement over the baseline configuration, enabled solely by a KV cache optimization trick.
Why It Matters
This workflow provides a roadmap for achieving high-performance, long-context inference on widely available, low-cost consumer hardware. It offers a practical method to run a capable dense model with multi-modal features (vision, audio, reasoning) without relying on expensive enterprise GPUs or cloud APIs, directly impacting cost reduction for developers and small businesses.
Practical Steps to Replicate the Performance
This workflow uses a specific fork of llama.cpp called TurboQuant to enable advanced KV cache quantization. The core achievement is running the entire 12B dense model on the GPU (-ngl 99) on just 8GB of VRAM, with a large context window.
1. **Get the Model:** Obtain the quantized Gemma 4 12B QAT model file. The example uses: `gemma-4-12B-it-qat-UD-Q4_K_XL.gguf`.
2. **Use the Correct Software:** You need the TurboQuant fork of llama.cpp, developed by Tom Turney (@no_stp_on_snek). The standard llama.cpp does not support the `turbo3` cache type.
3. **Launch with Specific Flags:** Run the model with the command that includes the crucial KV cache optimization flags. This is the key to freeing up VRAM for the larger context. The exact command provided is: `llama.cpp -m gemma-4-12B-it-qat-UD-Q4_K_XL.gguf -c 120000 --cache-type-k q8_0 --cache-type-v turbo3 -ngl 99 --port 8080`
4. **Understand the Result:** The flags `--cache-type-k q8_0 --cache-type-v turbo3` optimize the KV cache memory footprint. Combined with `-ngl 99` to offload all layers to the GPU, this allows the 120k context (`-c 120000`) to fit in 8GB of VRAM, achieving the reported high prefill and decode speeds.
