How to reduce LLM API costs in production by routing

AI strategy & adoption

The fastest way to reduce LLM API costs in production is to stop paying frontier rates for requests a cheaper model can handle: route each request to the cheapest model that can handle it. If your LLM application is past the pilot stage, your model bill moves in a straight line with usage. Every new user, every agent step and every retry adds tokens, and you pay frontier rates whether the request needed frontier reasoning or not. Routing is the one lever that moves the bill structurally. Here is the reference architecture: three routing patterns, cost figures you can check against AWS's published rates, a decision framework for where to start, and the failure mode that quietly undoes naive cost cutting.

How to reduce LLM API costs in production: the short answer

Route each request to the cheapest model that can handle it, and let a gateway make that call per request. Caching, batch APIs, output-length limits and enterprise discounts all bring the price per token down; routing changes which tokens you buy in the first place. The work is three moves: split traffic into two or three tiers by task class, ship static tiering wherever the task is known in code, then put a routing layer in front of the fleet for the traffic that enters through one interface. The rest of this article is how to do that without trading away quality.

Why does a single-model architecture tax every request?

In the UAE and across the GCC this has become a board-level line item rather than an engineering curiosity: Khaleej Times has reported on UAE firms facing rising AI costs as adoption moves beyond the pilot stage. The pattern is a familiar one: consumption compounds before the controls to manage it are in place, and the default habit of sending every request to the largest available model inflates cost per token across the whole estate.

What most teams reach for first is real but capped: prompt caching, batch APIs, output-length limits, enterprise discounts. All of these reduce the price per token. Routing changes which tokens you buy at all. The price gap between tiers is large enough to dominate everything else. On the AWS Bedrock rates used later in this article, Claude 3.5 Sonnet costs 12 times Claude 3 Haiku per input token and 12 times per output token. If half of your traffic can be handled by the smaller model, no caching strategy comes close to the same saving.

The formal framing in a 2025 survey of routing strategies for LLM-based systems (arXiv 2502.00409) is useful because it tells you what you are optimising. Given a pool of candidate models, a router picks the model that maximises a quality score for that query subject to a cost constraint. Two consequences follow. First, routing is a quality problem as much as a cost problem, which is why silent quality regression (below) is the failure mode to design against. Second, the router is an external, budget-aware component: it is not the internal gating network of a Mixture-of-Experts model, and candidates are not limited to neural networks. A route can point at a workflow, a retrieval step or a human.

The three routing patterns, and when each pays

The survey splits every published approach into two families by when the decision is made: pre-generation routing (predict which model will do well) and post-generation routing, also called cascade routing (generate first, evaluate, escalate if the answer is weak). In production those families collapse into three patterns you actually build.

1. Static tiering by task class. The task is known at the call site, so no router is needed. A marketing content tool calls a summarisation endpoint for one module and a generation endpoint for another; an exam-marking pipeline always sends open-ended answers to the same grading path. The AWS multi-LLM routing post calls this static routing: distinct UI components or endpoints per task, each with its own model. Zero added latency, zero added cost per request, and models can be swapped per component without touching the rest of the system. The limitation is rigidity: a new task means a new component.

2. Pre-generation routing (classifier or semantic). Prompts enter through one interface and a routing layer decides. Two implementations dominate:

• LLM-assisted routing: a small classifier LLM labels the prompt (task type, domain or complexity) and the request is dispatched accordingly. The AWS sample uses Amazon Titan Text G1 Express to classify tutor questions as history or math, then routes to Claude 3 Haiku or Claude 3.5 Sonnet. Fine-grained, but it adds a full inference call to the critical path. In AWS's measurements classification took about 0.53 to 0.60 seconds.
• Semantic routing: the prompt is embedded and matched against reference prompts per category by vector similarity. No generation step. AWS measured classification at about 0.09 to 0.11 seconds, roughly five to six times faster than the classifier approach, at a fraction of the cost (their embedding model ran at $0.00002 per 1,000 input tokens versus $0.0002 for the classifier). The trade-off is coverage: the reference set has to represent what users actually ask, and it needs ongoing curation.

The Google Cloud developer guide to model routing names the constraint behind the choice: the "Router Latency Paradox". A component designed to cut latency and cost must itself be near-free, so semantic routing is the default for interactive traffic and LLM-based routing is reserved for genuinely subtle distinctions. The guide's worked stack routes across Gemini 2.5 Pro, Flash and Flash-Lite with the open-source semantic-router library, a declarative router.yaml and FastAPI on Cloud Run. AWS's own hybrid suggestion is worth stealing for large category spaces: semantic search first to pick a department, then a small focused classifier within it.

The NVIDIA blueprint for cost-efficient LLM routing shows the production shape of pattern 2: a reverse proxy built in Rust with NVIDIA Triton, OpenAI API compliant, which parses the request, runs a classification model, dispatches (their demo sends code generation to Llama Nemotron Super 49B, open QA to Llama 3 70B and prompt rewrites to Llama 3 8B) and proxies the answer back. It deploys on a V100-class GPU with 4 GB, source in the NVIDIA-AI-Blueprints/llm-router repository, metrics into Grafana.

3. Post-generation routing (cascade). Answer with the cheap model first, evaluate the answer (a small confidence model, an LLM judge, or agreement across repeated samples), escalate to the next tier if it fails. FrugalGPT, the canonical example in the survey, uses a DistilBERT regressor to estimate answer correctness and reports 59% to 98% cost savings at comparable accuracy. Cascades are the strongest lever when quality risk is concentrated in a minority of hard requests, and the weakest when most requests are hard: the survey is explicit that cascading "can incur higher latency and cost, since multiple LLMs may need to generate answers for a single query".

Worked example: 100,000 questions a day, priced out

This is the scenario published in the AWS routing post: an educational tutor assistant in US East (N. Virginia) receiving 50,000 history questions and 50,000 math questions per day. We follow it step by step, then add the one number AWS implies but does not state: the all-frontier baseline.

Step 1: fix the traffic and token assumptions. 30-day month, so 1.5 million history and 1.5 million math questions per month. Each answer uses 150 input tokens and 300 output tokens. Router overhead is 150 input tokens per question (classification or embedding).

Step 2: price the no-routing baseline. If every question goes to Claude 3.5 Sonnet at $0.003 per 1,000 input tokens and $0.015 per 1,000 output tokens: 450 million input tokens per month cost $1,350, and 900 million output tokens cost $13,500. Baseline: $14,850 per month. (Arithmetic on the per-token rates AWS publishes in the post.)

Step 3: price the tiered architecture. History questions go to Claude 3 Haiku ($0.00025 in, $0.00125 out per 1,000 tokens). Math stays on Claude 3.5 Sonnet. AWS's published result:

Step 4: add the router itself. From AWS's second table: LLM-assisted routing costs $188.90 per month ($90 for the Titan classifier plus $98.90 for Lambda and API Gateway). Semantic routing costs $107.90 per month ($9 for Titan Text Embeddings V2 plus the same $98.90).

Step 5: total and saving. $8,232.65 per month with LLM-assisted routing, $8,151.65 with semantic routing, against the $14,850 baseline. That is roughly 45% saved, about $6,600 to $6,700 per month, and the router overhead is under 2.5% of the tiered bill. Latency: history answers returned in about 0.25 seconds of generation, math in 2.3 to 2.7 seconds, plus 0.1 seconds of semantic classification or 0.5 to 0.6 seconds of LLM classification.

Two lessons hide in these numbers. First, the saving comes entirely from moving 50% of traffic one tier down; the router is just the mechanism. Second, the choice between a $9/month embedding router and a $90/month classifier router is not about that $81. It is about 500 ms of added latency on every request, and about which one classifies edge cases correctly.

Which routing pattern should you start with?

Two honest calibrations from the research. The survey reports approaches reaching 95% of GPT-4 quality at 20% of cost on RouterBench, and MixLLM reaching 97% of GPT-4 accuracy at 24% of the cost. Those are research distributions, not planning numbers. For a GCC enterprise with mixed production traffic, plan on 30% to 50% for a first tiering pass, and treat anything beyond 60% as a signal that quality is being traded somewhere you have not measured yet.

How do you catch the silent quality regression?

This is the failure mode that makes naive cost cutting dangerous. The bill goes down, dashboards look green, and the system quietly gets worse on exactly the queries that mattered most: the ambiguous ones, the edge cases, the requests from your most valuable users. Nobody files a ticket; they just stop trusting the output.

The evidence is consistent across the literature. Domain-based routers score 77% to 97% accuracy inside neat categories but fall to 53% on "other" queries in the MoDEM work cited by the survey. A text-to-SQL router that tried to pick the cheapest adequate model lost roughly six success points (55.2% versus 61.0%) for only a 1.4 times cost reduction. The opposite failure is just as instructive: a similarity-based router on the same task pushed 81% of queries to GPT-4o and saved almost nothing (1.1 times). A router that always picks the frontier model is decorative; a router that always picks the cheap model is a quality incident with good unit economics.

The controls that work in production:

1. Default to the strong model when uncertain. AWS's tutor demo does exactly this: if the classifier is unsure whether a question is history or math, it labels it math, the expensive route. Pay the premium on the tail, not the average.
2. Keep a frozen evaluation set per tier. Fifty to two hundred representative queries per route, including the hard tail, scored every week with the same rubric or grader. Alert on deltas, not on levels.
3. Shadow before cutover. Route live traffic but also answer 5% of it with the frontier model, compare, and only then shift volume. Two weeks of shadow data will show you exactly which category leaks quality.
4. Log every routing decision. The Google Cloud guide is blunt about it: record chosen route, similarity score, final model, latency and estimated cost per request. Watch the distribution. If 80% of traffic still goes to the top tier, the router is not earning its keep; if 80% goes to the cheapest tier, check the golden set.
5. Write the quality floor down. RouterBench formalises it as predicted performance minus cost, weighted by willingness to pay. Whatever the formula, decide in advance what quality you refuse to go below, and configure the router to escalate rather than optimise through it.
6. Degrade on budget, do not silently cap quality. When a team hits 80% of its monthly budget, switch it to a cheaper or local model with an audit trail, instead of refusing requests or quietly degrading everyone. Cost control and quality control should not ambush each other.

The reference architecture

The production stack is thin and layered:

1. One OpenAI-compatible endpoint that every application calls. Clients never name a provider model; they ask for a logical name such as "assistant".
2. A gateway holding routing rules, failover chains, API keys with roles, rate limits and per-team budgets, all in configuration rather than in application code.
3. A routing decision made per request: static rule first, then semantic topic matching, optionally a classifier for fine-grained cases.
4. A model pool of two or three tiers, mixing your own GPU servers (vLLM, SGLang, llama.cpp) with hosted providers, so sensitive topics stay in-house and overflow spills to the cloud.
5. Observability: Prometheus metrics and OpenTelemetry traces, spend and route distribution on one dashboard, cost priced at the rate that applied when each request ran.

Illustrative routing configuration (rules evaluated in order, first match wins, each route carrying its own failover chain):

frontend_models:
  assistant:                 # the only name applications are allowed to call

routing:
  - match: { topic: [finance, legal] }
    route: [self_hosted_llama70b, bedrock_claude_haiku]   # sensitive stays in-house
  - match: { prompt_tokens: { max: 500 }, task: rewrite }
    route: [bedrock_claude_haiku, self_hosted_llama8b]
  - match: { budget_used_pct: { min: 80 } }
    route: [self_hosted_llama8b]                          # degrade, with audit log
  - match: {}                                             # catch-all: real reasoning
    route: [bedrock_claude_sonnet, self_hosted_llama70b]

balance: cache_affinity      # keep vLLM/SGLang prompt caches warm

And the shape of a cascade, if pattern 3 is the right one (the confidence estimator is the hard part; FrugalGPT uses a small supervised model, and an LLM judge or a self-consistency check also work):

def answer(query: str) -> str:
    for model in ("claude-haiku", "claude-sonnet"):   # cheapest first
        resp = llm.generate(model, query)
        if confidence(resp) >= THRESHOLD[model]:
            return resp.text
    return resp.text          # the strongest model's answer stands

Azrty's FastLLM Proxy is the gateway layer of exactly this architecture: an OpenAI-compatible gateway in front of your own LLM servers and 80 hosted providers, with routing, budgets and access control in one place. We are a Dubai company, and our own team in the UAE does the deployment and support. If what you need is the whole platform underneath the gateway, that is our AI infrastructure service.

What it costs to build, and what to expect back

Running costs of the router are small. In the AWS example the entire routing layer is $107.90 to $188.90 per month against a tiered model bill of about $8,000. A gateway on your own infrastructure costs a small VM or container plus a database for the control plane. The NVIDIA blueprint needs only a V100-class GPU with 4 GB for the classification model.

Engineering effort is the real cost. Budget for: reference prompt sets per route (start with 5 to 10 examples per category and iterate on logged misroutes), an evaluation harness with a golden set per tier, fallback chains and retry logic, dashboards for route distribution and cost per request, and a shadow-run period. The Google Cloud guide's production checklist is a fair proxy for scope: treat the router as a mission-critical standalone service, log every decision, refine route utterances continuously, and plan fallbacks beyond a single default model.

Managed routing trades control for speed. Bedrock Intelligent Prompt Routing is live and claims up to 30% cost reduction, but it works within a single model family (Claude and Llama at the time of writing) and the routing logic is not yours to tune. Fine for a homogeneous estate. Not fine if your mix includes your own GPUs, which data-residency requirements often drive in this region.

Expected saving, honestly stated. Static tiering with half the traffic a tier down: about 45%, as in the worked example. Savings grow with usage, discounts do not. Tiering beats negotiation because the gap between tier prices is structural.

What to do next

If you take one action from this article, make it this: pull last month's token spend by task class and find out what share of it was answerable by a model a tier down. That share is your savings ceiling, and it is usually visible in an afternoon. Then:

1. Classify your traffic into two or three tiers and set the rule that anything requiring multi-step reasoning or domain nuance goes to the frontier tier by default.
2. Ship static tiering first, wherever the task class is known in code. It is free and it is where most of the money is.
3. Put a gateway in front of the fleet so routing, keys, budgets and failover live in configuration, and so every request is logged with its route, model and cost.
4. Add semantic routing for the chat traffic that enters through a single interface. Start with 5 to 10 reference prompts per topic.
5. Shadow for two weeks against the frontier model on a sample, with a frozen golden set per tier and a written quality floor.
6. Turn on per-team budgets and chargeback reporting now, before agent workloads multiply the requests per business interaction.
7. Only then consider a cascade, and only for the specific classes that fail the quality floor.

Cost per request is the only model metric that scales linearly with your success. Architecture is how you bend the line.

https://www.azrty.com/blog/how-to-reduce-llm-api-costs-in-production-by-routing

<a href="https://www.azrty.com/blog/how-to-reduce-llm-api-costs-in-production-by-routing">How to reduce LLM API costs in production by routing</a> (Azrty)