Time to First Token
also called TTFT
How long before anything appears on screen.
The delay between sending a request and receiving the first token of the response. Driven by prompt length, model size and queueing, not by how long the answer is.
TTFT is what makes a product feel fast, because it is the wait before any feedback. Once tokens start streaming, perceived speed is dominated by tokens per second.
It is the metric to instrument first, because it is what users experience as responsiveness and it is driven by things you control: prompt length, tool-call count before the first visible output, and skipping work on the critical path.
Streaming the answer you were not ready to show
A three-step pipeline shows nothing for twelve seconds.
Move the parts that must happen before any output off the critical path, or stream a status while they run. Perceived responsiveness changes completely even though total completion time is nearly identical, because the user sees something happening.
- docsLatency optimisation ↗
The four levers that matter: fewer output tokens, fewer requests, parallel independent work, and streaming. Also defines throughput in tokens per second.
- docsPricing ↗
The current per-token and per-service prices. The right kind of source for token economics — and the reason no cost figure should be hard-coded without an access date.
Latency
Total time from request to finished answer.
Throughput
How many tokens per second the model generates.
Input Tokens · Prompt tokens
What you send. Cheaper, but repeated on every request.
Prefix Cache · Prompt caching
Reusing the model’s internal work for a prefix it has already seen.