Inference
Using a trained model — the part you pay for on every request.
Running a trained model to produce output. No weights change; the model just performs the forward pass and generates. This is what happens on every single API call.
Inference cost is driven by how many tokens go in and how many come out, plus how much hardware the model needs. It is the recurring bill, as opposed to training’s one-off one.
Every cost and latency conversation is an inference conversation. The levers are few and mostly mechanical: generate fewer tokens, make fewer calls, parallelise what is independent, reuse prefixes, and cache results you have already computed. Model choice is the fifth lever and usually the bluntest.
Cutting a slow feature without changing models
A three-step pipeline feels sluggish.
Independent steps run in parallel instead of in sequence, the shared instruction preamble moves to a cached prefix, and the responses stream so the user sees tokens immediately. Same model, same quality, materially different experience.
- docsLatency optimisation ↗
The four levers that matter: fewer output tokens, fewer requests, parallel independent work, and streaming. Also defines throughput in tokens per second.
- docsPrompt caching ↗
Caching reuses the model’s key-value state for a matching prompt prefix. Explains why reordering your system prompt throws away the cache.
- docsBatch processing ↗
Message batches for high-volume asynchronous work — evals, classification, backfills — with the submission and results lifecycle.
- docsBatch API ↗
Asynchronous batch requests for evals, classification and embeddings, with its own turnaround window and pricing — the cheap lane for non-interactive work.
Training
The expensive one-off process that sets the model’s parameters.
Token
The chunk of text the model actually sees — and the unit you are billed in.
Time to First Token · TTFT
How long before anything appears on screen.
Cost per Task
What one unit of real work actually costs.