LLM Systems
LLM Gateways and Routing
Why every serious LLM deployment ends up behind a gateway, and how to choose between LiteLLM, Portkey, OpenRouter, and rolling your own.
intermediate · 9 min read · Premium
You start with one provider key in an environment variable. Six months later you have four providers, fifteen models, three internal teams sharing the bill, two compliance reviews open, and an outage every time Anthropic's us-east region degrades. The LLM gateway is the piece of infrastructure that turns "we have an API key" into "we have a managed LLM platform." Either you build one or you run one of the open options.
What a gateway does that direct SDK calls do not
| Capability | Why you need it | Direct SDK |
|---|---|---|
| Unified API across providers | OpenAI, Anthropic, Groq, Gemini, Bedrock, Vertex all look the same to your app | Per-provider SDK each |
| Rate-limit handling | Provider 429s become routed retries instead of user-facing errors | Manual try/except per call site |
| Failover and fallback | Anthropic down -> Bedrock Claude -> OpenAI; transparent to the caller | None |
| Cost attribution | Per-team, per-user, per-feature spend, with budgets | Read provider dashboard, manually allocate |
| Policy-based routing | "Cheap model first, escalate on low-confidence" | Hand-written if statements |
| Per-tenant quotas | Free vs paid user limits enforced at the edge | Implement in your app, get it wrong |
| Audit log | Every prompt and response, indexed, exportable | None; you wire your own |
| Centralised observability | One trace UI for every LLM call, not one per provider | Per-provider dashboards |
The single biggest argument for a gateway: when you have N apps and M models, you have N*M wiring problems. A gateway makes it N + M.
The open options
LiteLLM (BerriAI). Python SDK + standalone proxy server. Unifies 100+ providers behind the OpenAI-compatible Chat Completions API. The proxy supports virtual keys per team/user, spend tracking, budgets, fallbacks, retries, and a usable admin UI. MIT-licensed, the de-facto default for self-hosted gateways. Easiest path: run the proxy in a container, point all your apps at it, give each team a virtual key.
Portkey. Hosted (with a self-host option) gateway with strong observability and prompt-management UX. Routes through 250+ models, ships caching, guardrails, and a prompt library. Picks itself when the team wants a dashboard rather than a config file.
OpenRouter. A unified billing layer in front of every model. Not really self-hosted - it is a marketplace. One API key, you can call Claude, GPT, Llama, Mistral, anything. The pitch is operational simplicity at the cost of putting your traffic through a third party. Strong for prototyping, agent development, and teams that want one invoice instead of twelve.
Cloudflare AI Gateway. Edge-deployed gateway with built-in caching and analytics. Picks itself if you are already on Cloudflare and want zero infra.
Keep reading with Pro.
You're reading the preview. Unlock the full concept plus the library, study plans, the AI mentor, and daily emails.