AI Gateway
also called LLM gateway, LLM proxy, Model gateway
One endpoint in front of every model provider, with routing, fallbacks and billing.
A proxy layer that sits between an application and one or more model providers, exposing a single API for many models. Instead of integrating with each provider separately, you call the gateway, which handles authentication, request translation, routing to a provider, retries and fallbacks, caching, rate limits, spend tracking and logging.
The gateway is where a multi-model strategy becomes operational: switching models is a config change, not a rewrite, and an outage at one provider becomes a fallback rather than an incident. The trade is a new dependency in the path of every request — the gateway sees your prompts, adds latency, and its pricing has to be weighed against the providers’ own rates.
The question that decides whether a gateway earns its place is ownership: who owns routing logic, outages, key rotation and cost reporting — you, the gateway, or both. Managed aggregators sell access and billing; self-hosted proxies run on your keys; edge gateways add caching and analytics.
Gateways come in three shapes: managed aggregators that sell access and billing, self-hosted proxies you run with your own keys, and edge gateways that add caching and analytics. Coding-agent-focused gateways add a fourth twist: they pre-test and benchmark the models they list for agent workloads, as OpenCode’s Zen does — the gateway is also a curation layer.
One outage, no page
A production agent calls a single provider directly and that provider has an outage.
With a gateway, the same request falls back to another provider serving the same model or a configured substitute, and the team reads about the outage in the gateway’s status log instead of in a user report. The fallback only works if it was configured and tested before the outage — a gateway does not invent one.
Assuming the gateway is free because the models behind it are cheap. Credit fees, markups and the cost of a middleman outage are the real price.
- guideThe LLM Gateway & Router Index (2026) ↗
A neutral matrix of gateway types — aggregators, proxy gateways, smart routers — with hosting, licence and best-fit for each.
- guideAI API Gateway 2026: 7 LLM Routing and Fallback Options ↗
Feature comparison across gateways; the useful row is ownership — who owns routing logic, outages, key rotation and cost reporting.
- docsOpenCode Zen ↗
The gateway’s own description: a curated set of models OpenCode has tested and benchmarked specifically for coding agents, usable with any agent.
Model Routing
Sending each request to the cheapest model that can handle it.
Model Provider
The company or platform you call to run inference.
Harness · Scaffold
Everything around the model that makes it an agent.
MCP · Model Context Protocol
An open protocol for plugging tools and data into any agent.
Rate Limit
The cap on how much you may send, per minute or per day.