Output Tokens
also called Completion tokens
What the model writes. Pricier, and the main driver of latency.
The tokens the model generates. Billed at a higher rate than input and produced one at a time, so they dominate how long a response takes.
If responses feel slow, count the output tokens. Time to answer is roughly output length divided by the tokens-per-second rate, on top of a fixed start-up delay.
Output tokens are where latency is decided, because they are generated one at a time. A response that is twice as long is roughly twice as slow to finish, which is why "be concise" is a performance instruction as much as a stylistic one.
Asking for the answer, not the essay
A reasoning model takes 40 seconds to respond to a simple question.
The thinking is billed and it is slow. Where the API allows it, cap the thinking budget and ask for the conclusion rather than the full working. Users perceive the difference immediately even though the answer is identical.
- docsLatency optimisation ↗
The four levers that matter: fewer output tokens, fewer requests, parallel independent work, and streaming. Also defines throughput in tokens per second.
- docsExtended thinking ↗
Configurable thinking budgets and interleaved reasoning around tool results — the API-level view of chain of thought as a resource you pay for.
Input Tokens · Prompt tokens
What you send. Cheaper, but repeated on every request.
Token
The chunk of text the model actually sees — and the unit you are billed in.
Latency
Total time from request to finished answer.
Throughput
How many tokens per second the model generates.
Time to First Token · TTFT
How long before anything appears on screen.