Speculative Decoding
A small model guesses ahead; the big model checks the guesses.
A decoding optimisation where a small draft model proposes several tokens and the large model verifies them in a single pass, accepting the ones it agrees with. The output distribution matches the large model while fewer expensive forward passes are needed.
It is one of the few inference optimisations that is close to free in quality terms, because verification guarantees the same distribution rather than approximating it. The win depends on how often the draft model guesses correctly, which is why it helps most on predictable text.
The quality story is what makes it attractive: verification preserves the target model’s output distribution rather than approximating it, so this is not a quality-for-speed trade like quantisation or a smaller model. The speed-up depends on how predictable the text is.
Where it helps and where it does not
Enabling speculative decoding on two workloads.
On structured, formulaic output the draft model predicts well and the gain is large. On creative, high-entropy text the draft is rejected constantly and you have added overhead. Measure both before making it a global setting.
- paperFast inference from Transformers via speculative decoding ↗
A small draft model proposes several tokens and the large model verifies them in one pass, preserving the target distribution while cutting the number of expensive forward passes.
- docsLatency optimisation ↗
The four levers that matter: fewer output tokens, fewer requests, parallel independent work, and streaming. Also defines throughput in tokens per second.
Inference
Using a trained model — the part you pay for on every request.
Latency
Total time from request to finished answer.
Throughput
How many tokens per second the model generates.
Sampling
How the model picks one token out of the distribution.
Quantization
Storing weights at lower precision to make a model cheaper to run.