Run a 12B Agentic Model Locally on an 8GB GPU Using an Optimized llama.cpp Fork
A user shares a step-by-step guide to running a powerful, fine-tuned coding and agentic model entirely on a single consumer GPU, eliminating API costs with optimized VRAM usage.
Practical Summary
This tutorial demonstrates how to run a 12-billion-parameter agentic model locally on a single 8GB VRAM GPU using a specialized fork of llama.cpp with TurboQuant, achieving 30 tokens per second and enabling complex agentic tasks without cloud API costs.
Why It Matters
For developers and small teams, this approach provides a direct path to eliminate recurring API costs for agentic AI tasks. It leverages open-source tools to run capable models on affordable hardware, offering control over privacy and workflow without subscription fees.
Step 1: Understand the Setup
The goal is to run the 'gemma-4-12B-agentic-fable5-composer2.5 V2' model, which is fine-tuned for coding and agentic tasks, locally on your machine. This removes reliance on paid cloud APIs. The author reports a major jump in agentic benchmark performance (from ~15% to ~55%).
Step 2: Check Hardware Requirements
A single NVIDIA RTX 4060 with 8GB of VRAM is sufficient. The key is using an optimized inference engine and model quantization to fit the model and its context into memory.
Step 3: Use the Optimized llama.cpp Fork with TurboQuant
Standard llama.cpp may not support all optimizations. The author uses a fork that includes TurboQuant, a method for compressing the key-value (KV) cache during inference. This allows for a 25,000-token context window on limited VRAM.
Step 4: Execute the Server Command
Run the following command to start the local model server. The flags enable full GPU offload (-ngl 99), set a 25K context (-c 25000), and apply mixed-precision KV cache compression (--cache-type-k q8_0 --cache-type-v turbo3) to minimize VRAM usage.
llama-server.exe -m gemma4-v2-Q4_K_M.gguf -ngl 99 -c 25000 --cache-type-k q8_0 --cache-type-v turbo3 --port 8080
Step 5: Access and Use the Model
Once the server is running, you can interact with it via its local port (8080). The model is designed for agentic workflows like writing code, running commands, and debugging, all running on your own hardware.