LLM Gateway Routing Policies: Route by Capability, Risk, and Budget, Not Random Model Selection

Once an AI product integrates multiple models, teams can easily reduce “routing” to weighted round-robin distribution: send 70% of requests to the primary model and 30% to a cheaper model, then automatically switch to the next model if the primary one fails. This approach may pass a demo, but in production it can send regulated data to the wrong region, route requests that require tool calls to models without structured output support, or repeat actions that have already produced side effects after an upstream timeout. The core responsibility of an LLM gateway is not to select an available endpoint, but to execute an auditable policy decision before a request enters the inference infrastructure.

A complete routing decision must read at least five categories of facts: the capabilities required by the request, data and tenant boundaries, task risk, the end-to-end latency budget, and the remaining cost budget. The model catalog provides context limits, tool-calling and multimodal capabilities, deployment regions, pricing, release versions, concurrency headroom, and current health status. Only backends that satisfy the hard constraints may enter the candidate set. Weights, prices, and real-time load may rank candidates only within that set; they cannot override compliance and capability restrictions.

Separate Hard Constraints from Optimization Objectives

Hard constraints are best enforced by deterministic code. For example, a request containing private documents from an EU tenant may be routed only to approved regions. A task requiring strict JSON Schema output must exclude versions that do not support it. High-risk write operations must use models that have passed the specified evaluation threshold. If no candidate satisfies every requirement, the gateway should return an explicit policy rejection or require the task to be narrowed, rather than silently relaxing the restrictions.

Optimization objectives determine which legal candidate to select. Interactive completion can prioritize time to first token, background summarization can prioritize cost per token, and complex code review can prioritize quality scores. The score can be expressed as qualityWeight * evalScore - latencyPenalty - costPenalty - saturationPenalty, but every input must have a version and collection timestamp. Do not let a model decide routing from a natural-language prompt because it cannot reliably know real-time capacity and cannot readily explain why a particular tenant was sent to a particular provider.

Routing configuration should be reviewed and versioned like code:

policy: interactive-code-v4
match:
  task: code_completion
  dataRegion: eu
require:
  toolCalling: true
  approvedRegions: [eu-west]
  minimumEvalScore: 0.86
optimize:
  firstTokenP95Ms: 900
  maxEstimatedCostUsd: 0.04
fallback: [same-model-other-zone, approved-secondary-model]

Before a call, estimate cost conservatively from the input tokens, reserved output limit, and expected cache hits, then update the actual value during streaming. This budget design can be combined with Token Cost Optimization and LLM Rate Limiting and Quotas so that policy failures are discovered before the month-end bill arrives.

Health Status Is More Than HTTP 200

A live model endpoint is not necessarily ready to receive traffic. The gateway must distinguish network reachability, queue saturation, time-to-first-token timeouts, output error rates, structured-result compliance rates, and content-safety anomalies. Health scores should be isolated by model version and region, with short windows detecting failures and long windows suppressing flapping. After a circuit breaker trips, recovery should use a small number of probe requests instead of restoring all traffic as soon as the cooldown expires.

Retries and fallback must account for request semantics. Plain-text generation can switch backends before any token has been returned; switching after streaming begins produces a stitched-together answer. A request with tool side effects may already have created a ticket or initiated a payment, so it must not be replayed blindly merely because the upstream response was lost. The gateway should propagate idempotency keys, record the invocation stage, and define explicit categories for “retryable,” “retryable only on the same version,” and “no retries.” Multi-Model Failover can serve as the foundation for the failure tree, but every fallback model must still be checked again against capability and data-boundary requirements.

Canary-Release the Routing Rules Themselves

Policy changes affect quality, cost, and capacity simultaneously. First replay historical de-identified requests offline and compare the candidate sets and rejection reasons produced by the old and new policies. Then enter shadow mode, calculating the new decision without actually shifting traffic. After confirming that there is no cross-region routing, missing capability, or budget violation, gradually expand the rollout through internal tenants, low-risk tasks, and percentage-based traffic stages.

Each decision should record at least requestClass, tenantIdHash, policyVersion, the candidates and exclusion reasons, the final backend, model version, budget estimate, fallback count, and terminal status. Do not record complete prompts by default. Dashboards should show routing hit rate, policy rejection rate, P95 queue and time-to-first-token latency for each backend, per-task cost, quality spot checks, and circuit-breaker counts. Observability fields can follow LLM Observability, while traffic allocation can draw from Load-Balancing Strategies, but the least-connections algorithm designed only for stateless HTTP services cannot be copied directly.

Rollback conditions must be fixed before release: roll back immediately after even one cross-region or unauthorized route. Pause the rollout if critical-task evaluations fall beyond the threshold, the policy rejection rate rises abnormally, or P95 latency or per-task cost repeatedly exceeds its limit. A rollback should switch only the policy version and retain the new version's decision logs, making it possible to determine whether the cause was a configuration error, stale model-health data, or inaccurate capacity estimates.

Validate the Policy with Failure Drills

Before launch, deliberately simulate a 429 response from the primary provider, a single-region network outage, an incorrect model-capability catalog entry, missing price data, a midstream disconnection, and a false health-probe result. For each case, confirm whether the request reaches the expected backend, whether tenant and regional constraints remain intact, whether a no-retry task is executed more than once, and whether the caller receives an explainable error. Then verify that when the configuration service is unavailable, the gateway uses the last signed policy rather than accepting an empty configuration or failing open by default.

A mature LLM gateway treats routing as a platform control plane: the data plane executes only validated policies, while the control plane manages the catalog, metrics, canary releases, and audits. When models are added, prices change, or a region fails, the team adjusts a bounded decision system instead of improvising weight changes during an incident.

Further Reading

• Multi-Model Failover
• LLM Rate Limiting and Quota Design
• LLM Observability
• Load-Balancing Strategies