LLM Model Routing to Keep Inference Costs Predictable

Opaeron, the operations platform I build at Trilliant Digital, has 711 API route handlers across 89 routers and 112 agent module files covering eleven marketing disciplines. Early on, every one of those agents that needed a language model named the model itself, inline, in the call site. That meant a classifier deciding whether a meeting line was a task was paying the same per-token rate as the agent writing a client-facing quarterly brief, and nobody could answer the question finance actually asked: what does one brief cost to produce?

#Why one model everywhere gets expensive

The default failure mode is not overspending on the frontier. It is spending frontier-tier money on jobs that never needed it. Inside Opaeron the AI work splits into shapes with very different difficulty: classification (which discipline does this request belong to, is this line a task or a note), extraction (turn a meeting transcript into structured tasks with owners and dates), generation (chat replies, executive briefs, ad copy), and embeddings (semantic search over historical account work, stored in a self-hosted PostgreSQL 16 instance running the pgvector/pgvector:pg16 image). Those four shapes differ by more than an order of magnitude in how much model they need, but a hardcoded model id treats them identically.

The market context makes this worth fixing rather than tolerating. Menlo Ventures put enterprise generative AI spend at $37 billion in 2025, up 3.2x from $11.5 billion in 2024, with foundation model APIs alone taking $12.5 billion of the $18 billion infrastructure layer. Their mid-year update found API spend more than doubling in six months, from $3.5 billion in Q4 2024 to $8.4 billion by mid-2025. The same survey found only 11% of teams switched vendors at all, while 66% upgraded within their existing provider. That inertia is mostly architectural: when the model id is scattered through the code, switching is a migration, so teams do not switch. A gateway makes the switch a one-line edit, which is the real reason to build one.

#The price table behind the router

Routing decisions are only as good as the numbers behind them, so the catalog carries published rates, not vibes. All AI traffic in Opaeron goes out through OpenRouter as a single gateway, but the pricing that drives routing comes from each provider's own published table.

Two details in those tables change routing decisions rather than merely describing them. Google prices Gemini 2.5 Flash-Lite at 3x cheaper input and roughly 6x cheaper output than Gemini 2.5 Flash, which is why the output column, not the input column, decides where classification runs. And on Gemini 2.5 Flash, Google prices audio input separately at $1.00 per 1M tokens against $0.30 (Flash-Lite: $0.30 against $0.10) for text, image and video, which is the reason meeting audio is always transcribed first and never handed to the model as audio. Batch modes cut both providers roughly in half (Google Batch at $0.15 / $1.25, Anthropic Batch at $0.50 / $2.50), which suits the nightly Celery beat jobs and not the interactive paths.

#How the router picks a model

The rule is simple to state and slightly annoying to enforce: no call site names a model. Call sites name a task. The gateway owns the mapping from task to an ordered list of candidate models, cheapest-that-clears-the-bar first. This lives in one module, and the LangGraph nodes that orchestrate the more involved agents call into it exactly like the plain FastAPI handlers do.

from dataclasses import dataclass

@dataclass(frozen=True)
class ModelSpec:
    slug: str
    usd_in_per_mtok: float
    usd_out_per_mtok: float

CATALOG = {
    "flash_lite": ModelSpec("google/gemini-2.5-flash-lite", 0.10, 0.40),
    "gpt_mini":   ModelSpec("openai/gpt-4o-mini",           0.15, 0.60),
    "flash":      ModelSpec("google/gemini-2.5-flash",      0.30, 2.50),
    "haiku_45":   ModelSpec("anthropic/claude-haiku-4.5",   1.00, 5.00),
}

# task -> ordered candidates. Index 0 is the cheapest model that has
# passed the frozen eval set for that task. The rest are failover only.
ROUTES = {
    "classify_discipline": ["flash_lite", "gpt_mini"],
    "task_or_note":        ["flash_lite", "gpt_mini"],
    "chat_reply":          ["flash", "gpt_mini", "haiku_45"],
    "exec_brief":          ["flash", "haiku_45"],
    "meeting_extract":     ["haiku_45", "flash", "gpt_mini"],
}

def candidates(task: str) -> list[ModelSpec]:
    try:
        return [CATALOG[key] for key in ROUTES[task]]
    except KeyError as exc:
        raise UnknownTask(task) from exc

Meeting extraction is the one task that starts on the most expensive model in the list, and that is deliberate. It has to return a strict Pydantic v2 schema (task title, owner, due date, account, confidence) over a noisy human transcript, and a retry plus a human correction costs far more than the difference between $0.30 and $1.00 per million input tokens. Cheap where the job is easy, expensive where a mistake is expensive, is the whole policy.

#Fallback chains when a provider fails

A single-provider setup fails whenever that provider fails. The ordered candidate list doubles as a failover chain, and the important constraint is that every entry has to be genuinely interchangeable: no provider-specific tool-calling schemas, the output contract expressed in the prompt and validated by Pydantic, and temperature and max-token settings normalised in the gateway. If the prompt only works on one provider, the fallback is decorative.

class RouteExhausted(RuntimeError):
    pass

RETRYABLE = (httpx.ReadTimeout, httpx.ConnectError, ProviderOverloaded)

async def complete(task, messages, *, model_cls=None, run_id=None, account_id=None):
    """Try each candidate in order. Cost is recorded per attempt, not per success."""
    last_error = None

    for spec in candidates(task):
        try:
            raw = await _post_openrouter(spec.slug, messages, timeout=45.0)
        except RETRYABLE as err:
            last_error = err
            log.warning("route %s: %s failed (%s), falling through", task, spec.slug, err)
            continue

        usage = raw["usage"]
        await record_usage(run_id, account_id, task, spec, usage)
        text = text_guard.clean(raw["choices"][0]["message"]["content"])

        if model_cls is None:
            return text

        parsed = coerce_json(text, model_cls)
        if parsed is not None:
            return parsed

        last_error = ValueError(spec.slug + " returned unparseable JSON")
        log.warning("route %s: %s failed schema, falling through", task, spec.slug)

    raise RouteExhausted(task) from last_error

Note that record_usage runs before the schema check. An attempt that returns garbage still burns tokens, and a cost ledger that only counts successes will quietly under-report every time a cheap model starts drifting. The failed attempts are the early warning signal.

#JSON repair for structured output

Models that mostly return valid JSON still return prose-wrapped JSON, fenced JSON, trailing commas, and the occasional apology. Falling through to a more expensive model on the first parse failure is the wrong instinct, because most of these breakages are lexical and fixable locally at zero API cost. My repair ladder has three rungs and only the third one spends money.

• Rung one, local salvage: strip code fences, take the substring from the first opening brace to the last closing brace, remove trailing commas before a closing brace or bracket, then json.loads and validate with the Pydantic v2 model.
• Rung two, schema coercion: accept a single object where a list of one was expected, coerce date strings through the same parser the API uses, and drop unknown keys rather than rejecting the payload.
• Rung three, one repair call: send the broken text back to google/gemini-2.5-flash-lite with the schema and the instruction to return only valid JSON. At $0.10 in and $0.40 out per 1M tokens this is the cheapest thing in the stack, and it is far cheaper than re-running the original extraction on Haiku 4.5.

Only after rung three fails does the gateway fall through to the next model. One extra Opaeron-specific wrinkle: every generated string passes through backend/shared/text_guard.py, which rewrites marketing cliches and strips em dashes before anything reaches a user. That cleanup runs after parsing, never before, because rewriting text inside a JSON payload before it has been parsed is a good way to corrupt a valid response.

#Per-run cost tracking

Price per million tokens is not a number anyone in an agency can act on. Cost per completed job is. Opaeron runs 77 Postgres tables, all prefixed tdx_, accessed through psycopg3 with a connection pool and a hand-written query-builder shim rather than an ORM, so the usage ledger is just another table: task, model slug, prompt and completion tokens, computed USD cost, the account it belonged to, whether the attempt succeeded, and the run id that groups every model call made while producing one artefact.

SELECT task,
       model_slug,
       count(*) FILTER (WHERE ok)                    AS ok_calls,
       count(*) FILTER (WHERE NOT ok)                AS failed_calls,
       round(sum(cost_usd)::numeric, 4)              AS spend_usd,
       round((sum(cost_usd) / nullif(count(DISTINCT run_id), 0))::numeric, 5)
                                                     AS usd_per_run
FROM tdx_ai_usage_log
WHERE created_at >= now() - interval '30 days'
GROUP BY task, model_slug
ORDER BY spend_usd DESC;

The usd_per_run column is the one that ends arguments. It divides by distinct run ids, so a brief that quietly made a classification call, three generation calls and a repair call shows up as one job with one price. Failed attempts stay in the numerator, because they are real money. A weekly Celery beat job prunes telemetry so the table does not grow without bound, and the same beat schedule already handles deadline reminders every 15 minutes and the nightly Postgres to Google Sheets backup.

#Deciding when a cheaper model is good enough

The honest version of this is that you cannot decide from the price table. You decide from an eval set. For each task I freeze fifty to a hundred real inputs pulled from production, grade the incumbent model against a rubric, and store that as the baseline. A cheaper candidate gets promoted only if it clears the same bar on the identical set, and only for that one task, with the incumbent demoted to first fallback rather than deleted. Downgrade one task at a time and watch the failure counter for a week.

RouteLLM, the open-source router from LMSYS and UC Berkeley, is the best public evidence that this works at scale: over 85% cost reduction on MT Bench, 45% on MMLU and 35% on GSM8K while still reaching 95% of GPT-4 performance, with its best router hitting that MT Bench figure while sending only 14% of calls to GPT-4. The point is not the specific percentages, it is that the routers were trained and evaluated against an explicit quality threshold. No threshold, no routing decision.

Two more findings shape how aggressive I am willing to be. Stanford HAI's 2025 AI Index records the cost of querying a model at GPT-3.5-equivalent MMLU (64.8) falling from $20.00 per 1M tokens in November 2022 to $0.07 by October 2024, more than a 280-fold reduction, and a16z's LLMflation analysis puts the general rate at roughly 10x per year at fixed quality, with GPT-3-level quality dropping 1,000x in three years. Epoch AI measured declines between 9x and 900x per year depending on which milestone you track, about 40x per year at GPT-4 level performance on GPQA Diamond, while cautioning that the fastest drops in that range came from the most recent year and may not persist.

The sharpest framing comes from Gundlach, Lynch, Mertens and Thompson, who find that the price for a given level of benchmark performance falls roughly 5x to 10x per year while the price of running frontier models rises 3x to 18x per year as models get larger and reasoning tokens pile up. Holding quality constant gets cheaper every year. Chasing the frontier gets more expensive every year. A router is how you sit on the first curve for the 80% of your traffic that does not need the second.

One counterweight worth keeping in view: in LangChain's survey of 1,340 practitioners, about one third named quality as the primary blocker to putting agents in production, and cost was cited less often than in previous years, which they attribute to falling prices and better efficiency. So I do not pitch routing internally as cost-cutting. I pitch it as a quality floor with a cost side-effect: the router guarantees a known model handles a known task, the eval set guarantees that model was measured, and the ledger tells you what the guarantee costs.

#What I would do first

If you are retrofitting this into an existing codebase, the order matters more than the sophistication. From doing it inside a platform with 121,931 lines of Python across 541 files, this sequence caused the least disruption:

• Log usage before you route anything. One table, tokens and computed cost per attempt, grouped by a run id. You cannot argue about model choice without it.
• Centralise the model id next, with the existing model as the only entry for every task. This change is behaviour-neutral and makes every later change one line.
• Add the fallback chain third, and make sure your prompts are portable across providers before you claim you have failover.
• Repair JSON locally before you retry remotely. Fenced blocks and trailing commas do not justify a second API call.
• Downgrade one task at a time against a frozen eval set, starting with classification, where the output-token gap between tiers is largest and the correct answer is a short label.
• Re-check provider pricing and model lifecycle pages on a schedule. Rates move, and models get retired underneath routers that nobody has looked at in six months.