Prefix Cache
also called Prompt caching, KV cache reuse
Reusing the model’s internal work for a prefix it has already seen.
An optimisation that stores the model’s internal state for a shared prefix of a prompt, so repeated requests skip recomputing it. You pay less — often much less — for cached tokens.
It works best when the unchanging part comes first: a long system prompt or document at the top, varying user input at the end. Order matters, since caching stops at the first difference.
Caching works on exact prefixes, so the ordering of your request is now an architectural decision. Static material belongs at the front — instructions, tool definitions, reference documents — and anything that changes per request belongs at the back. Get the order wrong and you pay full price for identical work.
The timestamp that destroyed the cache
A cache hit rate of near zero despite an identical system prompt every time.
Someone interpolated the current date into the first line. Everything after it is a different prefix on every request, so nothing ever matches. Move volatile values to the end and the hit rate jumps without touching the prompt content.
- docsPrompt caching ↗
Caching reuses the model’s key-value state for a matching prompt prefix. Explains why reordering your system prompt throws away the cache.
- docsContext caching overview ↗
Caching a large, reused prefix so you stop paying full price to resend it — and the cache-lifetime and prefix-stability conditions that decide whether it hits.
Cache Tokens
Tokens billed at the cached rate — cheap reads, sometimes a write fee.
Input Tokens · Prompt tokens
What you send. Cheaper, but repeated on every request.
Cost per Task
What one unit of real work actually costs.
Latency
Total time from request to finished answer.