LLM gateway routing: budgets across multiple providers

AI infrastructure & software

LLM gateway routing is how you put budgets and multiple providers behind one OpenAI-compatible endpoint: clients change a single base URL, and the gateway authenticates the request, picks the provider, falls back when one fails and enforces spend before the money is gone. If your models are scattered across self-hosted GPUs, Azure OpenAI, Anthropic, AWS Bedrock and a direct OpenAI contract, every application is holding its own provider keys, its own retry logic and its own idea of what a call costs. This walkthrough configures the four jobs the gateway does on every request, with a proxy config you can paste.

A model gateway answers the routing, budget and multiple-provider problem in one place: applications call a single endpoint, and the gateway decides which provider serves each request and whether the budget allows it. Most organisations we speak to in the UAE arrive at the same shape before that: a customer service copilot calling Azure OpenAI, a marketing team on a direct Anthropic key, a data science group running Llama on its own GPU servers, and one legacy integration still pointed at AWS Bedrock. Each application handles authentication, retries and cost tracking differently, or not at all. The result is predictable: no single view of spend, provider keys sitting in application config, and a provider incident on a Thursday afternoon taking down whichever app had no fallback.

The gateway becomes the only endpoint your applications know. Clients keep the OpenAI SDK and change one value: the base URL. Behind that URL the gateway does four jobs on every request, and this article walks through each one with configuration you can paste. We use LiteLLM's gateway documentation as the reference implementation, because it is the most widely deployed open source gateway with these features and its config format is a useful common language. At the end we map the same capabilities to FastLLM Proxy, our own OpenAI-compatible gateway for organisations that want the whole layer self-hosted.

What the gateway does on every request

It is worth being precise about the request path, because every budget, rate limit and fallback rule hangs off one of these steps. LiteLLM's life of a request documentation describes the sequence as:

1. Authentication and budget checks. The Bearer token is a virtual key. The gateway checks it against a Redis or in-memory cache first, and falls back to PostgreSQL on a miss. The key's own budget is checked here, before any provider call.
2. Rate limiting. A parallel request limiter checks requests per minute (RPM) and tokens per minute (TPM) at four levels: global server, virtual key, user and team.
3. Routing. The router picks a deployment from the requested model group, applying the routing strategy, then load balancing, retries and fallbacks.
4. Provider translation. The request goes out in the target provider's format and comes back in OpenAI format. The client never sees Anthropic's or Bedrock's wire format.

After the response returns to the client, spend logging, rate-limit accounting and logging callbacks run as asynchronous background tasks, so no database write sits in the request path. Two state stores carry the operational load: PostgreSQL holds keys, teams and spend, and Redis holds the key cache and the rate-limit counters. Run two or more gateway instances behind a load balancer without Redis and each instance keeps its own counters, which means a team's rate limit becomes N times whatever you set. Redis is not optional in production; it is what makes the numbers mean what you think they mean.

One virtual key per team, not one shared provider key

The single highest-value change is replacing shared provider keys with virtual keys. Each key carries its own model allowlist, budget, rate limits, team and user attribution. Provider API keys live only in the gateway's config or environment, so a leaked application key costs you a blocked virtual key, not your OpenAI account.

Create a key per team and per application, not per person, for production services:

curl 'http://gateway.internal:4000/key/generate' \
  --header 'Authorization: Bearer $LITELLM_MASTER_KEY' \
  --header 'Content-Type: application/json' \
  --data-raw '{
    "models": ["chat-default", "chat-economy"],
    "team_id": "support-copilot",
    "max_budget": 150,
    "budget_duration": "30d",
    "tpm_limit": 90000,
    "rpm_limit": 600
  }'

Spend is then attributable per key, user, team and end user via /key/info, /user/info and /team/info, which is the data your finance team needs for chargeback and your platform team needs for capacity planning. Two settings are worth enabling from day one:

litellm_settings:
  upperbound_key_generate_params:
    max_budget: 500        # reject any key asking for more than $500
    budget_duration: "30d"
    rpm_limit: 1000
  default_key_generate_params:
    budget_duration: "30d"

Upper bounds mean a team admin cannot mint an uncapped key through self-service. Defaults mean a forgotten max_budget still produces a capped key. In GCC enterprises where AI spend is centralised but experimentation is federated, this pair is the difference between governed self-service and a monthly surprise invoice.

How an LLM gateway routes across multiple providers within budget

A model group is a set of deployments that share one model_name and are load balanced across. Routing decides which deployment serves each request, and the budget rules decide which deployments are even eligible. The load balancing documentation and the router documentation describe five built-in strategies:

Two practical notes. First, simple-shuffle is the right default, and latency-based-routing with lowest_latency_buffer: 0.5 is the right choice for a small number of regional deployments of the same model, because the buffer keeps you from hammering the single fastest endpoint. Second, if you want a latency-based strategy for your flagship chat model but weighted shuffle for everything else, use routing groups rather than two gateways:

router_settings:
  routing_strategy: simple-shuffle
  routing_groups:
    - group_name: latency-sensitive
      models: [chat-default]
      routing_strategy: latency-based-routing
      routing_strategy_args: { ttl: 3600 }

You can also alias model names (model_group_alias) so applications request chat-default today and chat-default-2 after a model upgrade, with no code change.

Ordered fallbacks: surviving a provider outage

Retries and fallbacks are different mechanisms, and conflating them is how teams end up with apps that retry a dead provider twelve times. In the router architecture, retries stay inside the model group that failed and fallbacks leave it for the next group in your configuration.

Within a group, set order on each deployment. Lower is tried first. Each order level gets its own retries before the router escalates to the next:

model_list:
  - model_name: chat-default
    litellm_params:
      model: azure/gpt-4o                  # your own Azure deployment name
      api_base: os.environ/AZURE_API_BASE
      api_key: os.environ/AZURE_API_KEY
      order: 1      # primary
  - model_name: chat-default
    litellm_params:
      model: anthropic/claude-3-5-sonnet
      api_key: os.environ/ANTHROPIC_API_KEY
      order: 2      # tried when order 1 fails

The same block covers Bedrock: model: bedrock/anthropic.claude-3-5-sonnet-...-v1:0 with your AWS credentials in the environment is another order level in the chain, and the router treats it like any other deployment. Substitute your own model and deployment identifiers throughout.

Three behaviours to design around:

• 429s cool a deployment down immediately. If all order: 1 deployments are on cooldown, retries go straight to order: 2 without waiting for the full fallback path. A rate-limited primary therefore degrades in milliseconds, not minutes.
• Cooldown thresholds are configurable. allowed_fails: 3 with cooldown_time: 30 (seconds) means a deployment is parked for 30 seconds after more than three failures in that window. If you set nothing, the router falls back to a short default cooldown, so set both explicitly.
• Context-length and content-policy errors have their own fallback rules (context_window_fallbacks), so a prompt that is too long does not burn your whole fallback chain before failing.

Set num_retries and timeout in router_settings so a hung provider cannot pin a connection for minutes. A 30 second timeout with two retries per order level is a reasonable starting point for interactive chat.

Budgets across providers, models and tags: enforced before dispatch

This is the piece most gateways get wrong: they tell you what you spent after the money is gone. Budget routing instead removes over-budget deployments from the routing pool before the request is dispatched.

Provider budgets use Redis to track spend per provider over windows like 1d, 30d or 1mo, reset automatically, and are the mechanism behind the worked example below. Two operational details matter:

• Multi-instance deployments need Redis configured in router_settings, or spend counters diverge per instance and a $150 budget becomes $150 per pod.
• You can watch the remaining budget. GET /provider/budgets returns spend, limit and reset time per provider, and the Prometheus metric litellm_provider_remaining_budget_metric{api_provider="azure"} reports remaining dollars, which is what you alert on at 80 percent.

Hard rate limits on deployments are a separate control. With optional_pre_call_checks: [enforce_model_rate_limits], the rpm, tpm, rpd and tpd values on a deployment become enforced limits that return 429 with a retry-after header before the provider is called. RPM and RPD are exact; TPM and TPD are best-effort, because token counts are only known after generation. For daily quotas that reset on the provider's schedule rather than midnight UTC, set daily_reset_time (for example "08:00").

Worked example: a support copilot through one bad week

The setup. A Dubai retail group runs a customer support copilot: 10,000 chat turns per working day. Each turn is roughly 1,500 input tokens and 400 output tokens. The primary path is an Azure OpenAI deployment, with Anthropic as a second order, and a self-hosted Llama 70B on the group's own GPU servers as the final fallback. Finance has agreed a hard cap of 150 USD per day on Azure.

Step 1: cost per turn. Using example contract rates of 3 USD per 1M input tokens and 15 USD per 1M output tokens (replace these with your own price sheet, or set input_cost_per_token and output_cost_per_token explicitly on the deployment):

• Input: 1,500 x 3 / 1,000,000 = 0.0045 USD
• Output: 400 x 15 / 1,000,000 = 0.0060 USD
• Per turn: 0.0105 USD, so 10,000 turns = 105 USD per day

A normal day sits at 105 USD against a 150 USD cap. Headroom is 45 USD, or about 4,285 turns.

Step 2: the campaign spike. A marketing campaign doubles support volume to 21,000 turns. If all of it went to Azure it would cost 220.50 USD. Instead, the provider budget trips after 150 / 0.0105 = 14,285 turns. From that moment the router removes Azure from the pool and the remaining 6,715 turns go to the order: 2 Anthropic deployment. Azure spend holds at roughly 150 USD, the copilot keeps answering, and your Monday report shows exactly which provider absorbed the overflow.

Step 3: the provider incident. On another day Azure returns 429 to every call at 10:15. The failing deployment is placed on cooldown immediately, retries move to order: 2 within the same request, and users see slightly higher latency rather than errors. If Anthropic also fails, the configured fallbacks entry routes to the self-hosted model group, where the marginal cost is GPU time you have already paid for.

Step 4: the accounting. The next morning, GET /provider/budgets and the team's key spend endpoint give you the numbers for the chargeback meeting: provider-level spend against cap, and per-team spend against each key's max_budget. No spreadsheet reconstruction required.

The configuration behind it

This is a minimal proxy_config.yaml that implements the scenario: model list, ordered deployments, routing, fallbacks, provider budgets, enforced rate limits, Redis and a master key. Substitute your own model and deployment names.

model_list:
  - model_name: chat-default
    litellm_params:
      model: azure/gpt-4o
      api_base: os.environ/AZURE_API_BASE
      api_key: os.environ/AZURE_API_KEY
      rpm: 600
      tpm: 90000
      order: 1
  - model_name: chat-default
    litellm_params:
      model: anthropic/claude-3-5-sonnet
      api_key: os.environ/ANTHROPIC_API_KEY
      rpm: 600
      order: 2
  - model_name: chat-economy
    litellm_params:
      model: openai/llama-3.3-70b-instruct   # your vLLM or TGI server, OpenAI-compatible
      api_base: http://vllm.internal:8000/v1
      api_key: os.environ/VLLM_KEY
      input_cost_per_token: 0.0000004  # GPU time amortised, per your own maths
      output_cost_per_token: 0.0000008

router_settings:
  routing_strategy: simple-shuffle
  num_retries: 2
  timeout: 30
  fallbacks:
    - chat-default:
        - chat-economy
  provider_budget_config:
    azure:
      budget_limit: 150
      time_period: "1d"
    anthropic:
      budget_limit: 300
      time_period: "1d"
  optional_pre_call_checks:
    - enforce_model_rate_limits
  redis_host: os.environ/REDIS_HOST
  redis_port: os.environ/REDIS_PORT
  redis_password: os.environ/REDIS_PASSWORD

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL

litellm_settings:
  upperbound_key_generate_params:
    max_budget: 500
    budget_duration: "30d"

Start it with litellm --config proxy_config.yaml, then point a client at it with no other code changes:

import openai

client = openai.OpenAI(
    api_key="sk-team-virtual-key",
    base_url="http://gateway.internal:4000",
)
response = client.chat.completions.create(
    model="chat-default",
    messages=[{"role": "user", "content": "Where is my order?"}],
)

Running the same layer on your own infrastructure

Everything above is a pattern, not a product mandate. The pattern is: one OpenAI-compatible endpoint, virtual keys per team, a router with ordered fallbacks, and budgets enforced before dispatch with Redis behind them. If your constraint is that the gateway itself must run inside your own data centre or cloud tenancy, with provider credentials never leaving your network, that is the problem FastLLM Proxy is built for. It is an OpenAI-compatible gateway that sits in front of your own LLM servers and 80 hosted providers, with routing, budgets and access control in one place, and it maps onto this article one to one: virtual keys for per-team access and spend, routing and ordered fallbacks across your GPU fleet and hosted providers, and per-provider budgets held on your own infrastructure.

For regional organisations the operational argument is simple. The gateway becomes the single place where data residency policy is enforced, because it is the only component that touches both your prompts and the provider APIs. Your applications hold no provider keys, prompts can be kept on self-hosted models where policy requires it and routed to hosted models where it does not, and the whole control layer runs on infrastructure your team operates. Our own team in the UAE does this work: we design the platform on AI infrastructure engagements, deploy the gateway and the GPU fleet behind it, and run them for you.

What to do next

If you take one thing from this article, take the order of operations. It matters more than the tool choice.

1. Inventory every provider key in application config and environment variables. Count them. That number is your current blast radius.
2. Stand up one gateway (LiteLLM or FastLLM Proxy) with PostgreSQL and Redis, in the environment where your data policy says prompts may flow.
3. Define model groups per capability (chat, embeddings, batch), not per provider, and give each group an order chain ending in something you control.
4. Issue one virtual key per team and application, with max_budget, budget_duration and RPM/TPM limits, and set upper bounds so self-service stays governed.
5. Set provider budgets (provider_budget_config) at the daily caps finance has actually agreed, and alert on litellm_provider_remaining_budget_metric at 80 percent.
6. Run one outage drill per quarter: block the primary provider at the firewall and confirm the fallback chain absorbs it within one request, not one incident call.

Do those six steps and the phrase "one gateway for every model" stops being a slide and becomes the operational reality your budget reports, your incident runbooks and your developers all rely on.

https://www.azrty.com/blog/llm-gateway-routing-budgets-across-multiple-providers

<a href="https://www.azrty.com/blog/llm-gateway-routing-budgets-across-multiple-providers">LLM gateway routing: budgets across multiple providers</a> (Azrty)