Rate Limit
The cap on how much you may send, per minute or per day.
Provider-enforced ceilings on requests or tokens over a time window. Exceeding them produces errors and delays, and they constrain scale differently from cost.
Rate limits are a launch risk: a feature that is cheap can still fall over when a hundred users hit it at once. Ask for the quota, and handle error responses with backoff and a fallback model.
Treat them as a scheduling constraint rather than an error condition. Requests per minute and tokens per minute are separate ceilings, so a workload with large prompts hits the token ceiling long before the request ceiling, and a queue with backoff is the standard answer.
The workload that hit the token ceiling
A job sending few requests, each with a huge document, starts failing.
The request rate is well within limits and the token rate is not. Recognising which ceiling applies tells you whether to slow down, shrink the payload, or move to the batch lane — and the error handling differs in each case.
- docsRate limits ↗
Requests per minute and tokens per minute as separate ceilings, and how they are applied per model, project and account. Values change; the structure does not.
- docsBatch processing ↗
Message batches for high-volume asynchronous work — evals, classification, backfills — with the submission and results lifecycle.
Model Routing
Sending each request to the cheapest model that can handle it.
Throughput
How many tokens per second the model generates.
Observability
Being able to see what the system actually did.
Vendor Lock-in
How hard it would be to leave your model provider.