Cost per Task
What one unit of real work actually costs.
The total token and infrastructure cost of completing one real task — a resolved ticket, a generated report, a merged pull request — rather than the price per token.
Cost per token is a distraction; cost per completed task is the number a business can judge. Retries and failed attempts are part of the cost, so reliability improvements pay for themselves.
Its value is that it makes architectures comparable. A cheaper model used three times can cost more than an expensive one used once; a cached prefix can cut an input-heavy flow dramatically; a batch lane can halve a non-interactive job. None of those comparisons are visible in a price per million tokens table.
The cheap model that cost more
A smaller model is used for extraction because it is cheaper per token.
It needs retries on the hard third of inputs and produces more review work, so cost per correctly extracted document is higher. Instrument cost per completed task, including retries, and the decision usually reverses.
The non-interactive half of the workload
An evaluation suite runs nightly over a few thousand cases.
There is no user waiting, so the asynchronous batch lane is the right home for it. Same models, same outputs, materially lower unit cost — and the queue is not competing with interactive traffic for rate limits.
- docsPricing ↗
The current per-token and per-service prices. The right kind of source for token economics — and the reason no cost figure should be hard-coded without an access date.
- docsBatch processing ↗
Message batches for high-volume asynchronous work — evals, classification, backfills — with the submission and results lifecycle.
- docsPrompt caching ↗
Caching reuses the model’s key-value state for a matching prompt prefix. Explains why reordering your system prompt throws away the cache.
Token
The chunk of text the model actually sees — and the unit you are billed in.
Input Tokens · Prompt tokens
What you send. Cheaper, but repeated on every request.
Output Tokens · Completion tokens
What the model writes. Pricier, and the main driver of latency.
ROI · Return on investment
Did it save more than it cost?
Model Routing
Sending each request to the cheapest model that can handle it.