Latency
Total time from request to finished answer.
End-to-end response time: the start-up delay plus generation time for every output token. The dominant determinant of whether a feature feels usable.
Latency is governed mostly by output length and model size. The cheapest wins are usually reducing output and routing easy work to a smaller, faster model.
The decomposition that helps in practice: start-up delay, plus output tokens multiplied by generation rate, plus anything dependent that ran in sequence. Most latency work is which of those three you attack — and shortening the answer is the one people forget.
Four levers, in order of payoff
A feature feels slow and the model choice is fixed.
Generate fewer tokens, make fewer calls, parallelise the independent ones, and reuse cached prefixes. Together those usually recover more than switching to a smaller model would, and the answer quality does not move.
- docsLatency optimisation ↗
The four levers that matter: fewer output tokens, fewer requests, parallel independent work, and streaming. Also defines throughput in tokens per second.
- docsPrompt caching ↗
Caching reuses the model’s key-value state for a matching prompt prefix. Explains why reordering your system prompt throws away the cache.
Time to First Token · TTFT
How long before anything appears on screen.
Throughput
How many tokens per second the model generates.
Output Tokens · Completion tokens
What the model writes. Pricier, and the main driver of latency.
Model Routing
Sending each request to the cheapest model that can handle it.