What is an LLM gateway, and do I need one?
Short answer
An LLM gateway is a single layer in your own stack that every model call passes through. Call sites name a task, such as classify or summarise, and the gateway decides which model serves it, retries or falls back to another provider on failure, and records tokens and cost for each call. You need one once more than a couple of features call models, because it turns a model change or a price check into a one-line edit instead of a code search.
#What a gateway does
- Routing: maps a task to an ordered list of models, cheapest that clears your quality bar first.
- Fallback: when a provider errors or times out, the next model in the list takes over, so one outage is not your outage.
- Cost tracking: writes tokens and computed cost for every attempt, grouped by a run id, so you can answer what one job costs.
- One place for the boring parts: timeouts, retries, output repair and token limits are written once.
#Build one or use a hosted or open-source gateway
Hosted marketplaces and open-source proxies both exist, and they differ mainly in who runs them, how fees are charged and how much governance they add. Compare current pricing and terms on each vendor's own pages before deciding, because they change often.
The design that matters is the same either way: no call site names a model. The worked example, with a price table, fallback chain and per-run ledger, is in LLM Model Routing to Keep Inference Costs Predictable.
#When you do not need one yet
If one feature calls one model and spend is small, a thin wrapper function with a usage log is enough. Add the gateway when you have a second provider, a second team, or a bill someone asks you to explain. Log usage first, route second.
Sources
- Dinesh Solanki, LLM Model Routing to Keep Inference Costs PredictableWorked example of task-based routing, fallback chains and per-run cost tracking, with provider price tables.
Read the full write-up
Have a question that is not answered here?
Ask me directly