Open-weight models capture 56% of gateway tokens as enterprises route by price

Here is the business reality: LLM inference has transitioned from a pure capability competition into a disciplined procurement and routing challenge. AT&T demonstrates the playbook clearly. Its internal platform scaled from 8 billion tokens a day to 45 billion, yet overall spend stayed flat because 25% to 40% of requests run on open models, heading towards 70%. The operational metric your CFO will scrutinise is not benchmark scores; it is the blended cost of a routine token and the architecture that determines where it executes. For GCC enterprises, this transition solves a major compliance barrier: running open-weight models on dedicated regional cloud instances or on-premises GPU clusters keeps regulated banking, telecom and government data strictly inside national borders while unit prices decline. Vercel recorded an average token price reduction of 23.2% in August 2026 alone, its third straight monthly drop.

Two operational nuances carry more weight than the headline metric. First, request volume and expenditure have severed their historical correlation: open-weight weights absorbed 56% of gateway tokens while generating only 14% of spend, proving frontier providers still retain margin on complex queries. Second, model migration is occurring internally within vendor catalogues rather than through outright churn: Fable 5 billings collapsed from 13.2% to 4.9% in one month as Opus 5, at half the price point, gained 22.5%. The underlying unit economics explain the migration: self-hosted inference ranges from $0.17 to $1.00 per million tokens, compared with $5 to $15 for hosted frontier APIs. Stanford's 2026 AI Index notes the leading proprietary system holds just a 3.3% lead over leading open models on standard benchmarks, while OpenRouter's rankings show DeepSeek handling 25.4% of text requests and OpenRouter's published data confirms discounting models expanded token volume by 13.8 times.

Most enterprise teams stumble on three specific implementation flaws. First, they switch models without a unified routing gateway and discard their prompt cache, which AT&T notes costs tenfold more when repopulated with raw tokens. Second, they misinterpret 56% open volume as justification to cancel frontier subscriptions, only to watch failure rates spike on the critical 10% of high-consequence edge cases. Third, they treat open weights as zero-cost software, ignoring evaluation harness development, fine-tuning and GPU provisioning. AT&T succeeded because it built domain fine-tunes of Gemma (OTel 1.0 and 2.0, trained on 15 billion telecommunications tokens) and scored routing paths continuously. In the GCC, the equivalent imperative is dialectal Arabic, legal precision and local regulatory corpora, where targeted fine-tunes outperform generalist frontier APIs at a fraction of the operating cost.

Technically, we treat this as an infrastructure and gateway challenge. Place an OpenAI-compatible gateway in front of all runtime environments: FastLLM Proxy centralises routing, quotas, budgets and role-based access across internal inference nodes and hosted commercial APIs. On the compute layer, host open weights via Kuvryn AI to run models, agents and GPU workloads across isolated multi-tenant clusters. Then implement deterministic routing policies: an economy tier on self-hosted instances, a frontier tier for verified high-complexity prompts, automated quality fallback triggers, cache affinity for multi-turn agent threads, and hard tenant quotas. Using patterns from the LiteLLM routing documentation, a production configuration follows this structure:

yaml\nrouter_settings:\n routing_strategy: cost-based-routing # economy tier first, escalate on quality gates\n optional_pre_call_checks: [\"session_affinity\"]\n deployment_affinity_ttl_seconds: 3600 # pin the prompt cache to one deployment\n\nmodel_list:\n - model_name: economy # bulk traffic: self-hosted open-weight\n litellm_params:\n model: openai/gemma-4 # OpenAI-compatible backend (vLLM, TensorRT-LLM)\n api_base: https://llm-gpu.internal:8000\n input_cost_per_token: 0.0000002 # $0.20 per million tokens\n output_cost_per_token: 0.0000004\n - model_name: frontier # escalation tier for complex requests\n litellm_params:\n model: anthropic/claude-opus-5\n api_key: os.environ/ANTHROPIC_API_KEY\n\nlitellm_settings:\n num_retries: 3\n