Train a compact model for budgeted reasoning to cut inference token costs by over 80%
A $8 project distilled a 32B teacher model into a 4B student, achieving near-peak accuracy with ~80% fewer reasoning tokens, demonstrating a practical workflow for reducing LLM operational costs.
Practical Summary
This workflow shows how to create a smaller, cost-efficient reasoning model by distilling knowledge from a larger one into explicit token budgets. The key result is that you can train a model to reason with drastically fewer tokens (e.g., 99 vs. 497) while maintaining high accuracy (88.8% vs. 93.6%), directly cutting inference costs.
Why It Matters
For businesses, this method translates to direct cost savings on API calls or self-hosted inference by reducing the number of tokens a model generates to solve a problem. It provides a reproducible pipeline for creating optimized, budget-aware AI models without prohibitive training costs.
Overview: Distilling a reasoning model for token efficiency
The core goal is to teach a smaller model to reason with explicit length budgets (full, lightly compressed, medium compressed, very concise) by learning from a larger teacher model. This directly reduces inference costs by generating fewer output tokens per query.
The project used Qwen3-32B as the teacher and Qwen3-4B as the student, trained on the GSM8K math dataset. The entire process was executed within a strict $8 compute budget.
Step-by-step pipeline for budgeted reasoning distillation
1. **Generate Teacher Traces:** Use the large teacher model (e.g., Qwen3-32B with vLLM) to generate verified reasoning traces for your dataset (like GSM8K).
2. **Create Budgeted Versions:** Rewrite each correct teacher trace into multiple versions at different compression levels (L1, L2, L3) while re-verifying the final answer's correctness. This creates your SFT (Supervised Fine-Tuning) data.
3. **Train the Student Model:** Fine-tune the compact student model (e.g., Qwen3-4B) on these budgeted traces using a method like LoRA SFT with bf16 precision.
4. **Evaluate for Cost-Accuracy Trade-off:** Test the trained adapters on a held-out set. Look for the 'Pareto optimal' points where you get the highest accuracy for the fewest tokens.
