Distillation
Training a small model to imitate a big one.
Training a smaller, cheaper model to reproduce the outputs or probabilities of a larger teacher model, transferring much of the capability into a fraction of the size.
Distillation is how many fast, cheap, domain-specific models are made. You pay the big model once to generate training data, then serve the small one.
Distillation is why small models keep getting better than their size suggests, and it has a second, less discussed use: you can distil a specific capability rather than everything. Take the traces from a frontier reasoning model on your task and train a small model to produce them, and you buy most of the quality for a fraction of the per-call price.
Distilling your own workload
Ninety percent of traffic is a narrow extraction task handled by a frontier model.
Run the expensive model over a sample, keep its outputs as training data, and distil a small model on that distribution. You keep the provider as the fallback for hard cases and cut unit cost on the bulk of traffic.
- paperDistilling the knowledge in a neural network ↗
The original distillation idea: train a small model to match a larger one’s soft outputs. Foundational for the concept, distinct from later LLM-specific student/teacher recipes.
- paperDeepSeek-R1: incentivizing reasoning capability in LLMs via reinforcement learning ↗
Reinforcement learning on verifiable tasks elicited reasoning patterns — self-checking, backtracking — with substantial open publication of the recipe.
Fine-tuning
Extra training on your own examples to change how the model behaves.
Quantization
Storing weights at lower precision to make a model cheaper to run.
Model Routing
Sending each request to the cheapest model that can handle it.
Cost per Task
What one unit of real work actually costs.