Local LLM Benchmark for Three.js Development: Choosing Between Speed and Code Quality
A developer's hands-on test comparing Gemma 4 12B QAT and Qwen 3.6 35B-A3B for generating Three.js code on a 12GB consumer GPU, revealing that faster inference doesn't always mean better practical output.
Practical Summary
A developer benchmarked two open-weight LLMs for a specific workflow: generating Three.js game code using a single RTX 3060 12GB GPU and llama.cpp. The faster Gemma 4 12B QAT model delivered ~50 tokens per second and fit entirely in VRAM, while the larger Qwen 3.6 35B-A3B model was slower (~25-30 tok/s) and required memory offloading. However, when evaluating on three real-world prompt tasks, Qwen consistently produced more accurate and usable code. The conclusion is that for code generation workflows, output quality can outweigh raw speed and memory efficiency, making the slower but more capable model the better practical choice.
Why It Matters
This real-world comparison provides a reproducible framework for developers evaluating local LLMs on consumer hardware. It highlights that workflow-specific benchmarks—testing on actual tasks like Three.js simulation prompts—are more valuable than generic leaderboards. The finding that a larger, slower model can deliver superior practical results helps teams make informed decisions about tool cost optimization, balancing compute time against the cost of editing or discarding low-quality generated code.
How to Benchmark a Local LLM for Your Specific Development Workflow
The author's approach provides a template for developers to test local models against their own use case. Instead of relying on public benchmarks, you can run a targeted comparison to see which model performs best for your tasks.
Step 1: Define a Clear Test Task. The author used '3 one shot Three.js simulation prompts.' These are specific, practical tasks that reflect real work. Choose prompts that represent your most common or critical development requests.
Step 2: Standardize the Inference Setup. For a fair comparison, control the environment. The author used llama.cpp with consistent settings: MTP enabled (n-max=4), Q8 KV cache, and identical parameters. This isolates model performance from software differences.
Step 3: Measure Both Performance and Quality. The author recorded two key metrics: inference speed (tokens per second) and output quality ('much closer to the prompt'). Speed is easy to measure; quality requires human evaluation of the generated code's correctness and usefulness.
Step 4: Account for Hardware Constraints. Note which models fit in your GPU's VRAM versus which require offloading. The author noted Gemma fit fully, while Qwen required offloading, impacting its speed. Your hardware will dictate viable model choices.