Throughput
How many tokens per second the model generates.
The generation rate, usually measured in output tokens per second. Higher means the same answer arrives sooner.
Throughput matters most for long outputs — reports, code, summaries. Streaming hides latency but not a slow generation rate.
Throughput and latency are different questions and optimising one can hurt the other. Batched requests improve throughput and worsen per-request latency; aggressive speculative decoding can do the reverse. Knowing which number your product is judged on is what makes the trade-off decidable.
Choosing the right number to optimise
A nightly backfill of a hundred thousand documents.
Nobody is waiting, so latency is irrelevant and cost per item plus total wall-clock are what matter. That is exactly the workload for the asynchronous batch lane, which trades turnaround time for a materially lower unit price.
- docsLatency optimisation ↗
The four levers that matter: fewer output tokens, fewer requests, parallel independent work, and streaming. Also defines throughput in tokens per second.
- paperFast inference from Transformers via speculative decoding ↗
A small draft model proposes several tokens and the large model verifies them in one pass, preserving the target distribution while cutting the number of expensive forward passes.
Latency
Total time from request to finished answer.
Output Tokens · Completion tokens
What the model writes. Pricier, and the main driver of latency.
Quantization
Storing weights at lower precision to make a model cheaper to run.
Rate Limit
The cap on how much you may send, per minute or per day.