LLM Gateway and Model Routing: A Decision Framework for Multi-Provider AI Systems

At a glance

Five stages a request moves through before any model sees it — skipping the last one is why routing bugs get misdiagnosed as model bugs.

This guide is for engineers and platform leads running production traffic across two or more LLM providers or models who need a defensible policy for which request goes to which model, and who need to debug when a routing decision — not the model itself — caused a quality or cost regression. It does not cover how to make a single provider relationship reliable; operating LLM APIs in production already covers timeouts, retries, and response validation for one call at a time. This article sits one layer above that: the decision about which call to make in the first place, across a portfolio of models, and how to keep that decision auditable.

The routing ledger model

Use Eligibility → Candidates → Signal → Fallback → Ledger as the decision path:

1. Eligibility: which providers are even legally and contractually allowed to see this request.
2. Candidates: which of the eligible models can actually do this class of task.
3. Signal: the specific rule — cost, latency, confidence, or task type — that picks one candidate for this request.
4. Fallback: the ordered chain the request follows when the chosen model or provider fails.
5. Ledger: the record of which model actually served the request, and why, written at request time rather than reconstructed later.

Decision rule: If you cannot answer "which model served request X, and why" for a request from six weeks ago, you do not have a routing policy — you have a load balancer with opinions nobody wrote down.

The framework is narrower than the whole subject of multi-model architecture. It does not cover prompt engineering per model, fine-tuning, or how to build an evaluation harness — a practical guide to AI agent evaluation covers the second half of that problem. It covers the decision layer that sits between "the request arrived" and "a specific model produced a specific answer."

1. Define eligibility before you define preference

Hard constraints come first

Eligibility is not the same question as "which model is best." It is "which models are allowed to touch this request at all." Data residency requirements can rule out a provider whose inference runs in a region your contract forbids. A safety or content-policy tier can rule out a provider that has weaker guardrails than a regulated workflow requires. A tool-calling requirement can rule out a model that does not support the function-calling schema your agent depends on. None of this is about quality — it is about whether the request is even permitted to reach a given provider.

Treat eligibility as a hard gate evaluated before any scoring happens. A common failure mode is folding eligibility into the same weighted-score function used for quality or cost, so a cheap ineligible model can still win on paper. If a model is ineligible, it should never enter the candidate set, regardless of how attractive its cost or latency profile looks.

Eligibility is a filter, not a scoring dimension. Mixing the two lets a disqualifying constraint get outvoted by an attractive price.

Contractual and compliance eligibility

Some eligibility constraints are static — a data processing agreement that names approved subprocessors, or a policy that certain document classes never leave a specific cloud region. Others are dynamic — a customer's plan tier might permit only "reviewed" providers, or a workflow flagged for a security incident might be temporarily restricted to a single trusted model while an investigation is open. Encode both kinds explicitly. A static allowlist per data classification, checked before any routing signal runs, prevents an eligibility violation from ever depending on a scoring bug.

Field note: Teams that skip explicit eligibility checks tend to discover the gap the same way — a proof-of-concept integration with a new provider gets promoted to production traffic before anyone confirms it is covered by the same data processing agreement as the providers already in rotation. The fix is cheap when it is a checklist item before rollout; it is expensive when it is a compliance incident after the fact.

2. Build the candidate set

Task-based model selection

Once eligibility narrows the field, the candidate set answers a different question: which of the remaining models are actually good at this kind of request? A model tuned for code generation may underperform on open-ended summarization, and a fast, inexpensive model may be entirely adequate for a classification task while being a poor fit for multi-step tool use. Task-based routing groups requests by the kind of work they represent — coding help, summarization, structured extraction, open-ended conversation, tool-calling workflows — and defines a candidate list per group rather than one global list for all traffic.

This is where most teams get the most leverage with the least risk. A global "cheapest model that passes eligibility" policy will save money on easy requests and quietly degrade quality on hard ones, because it never asks what kind of task it is looking at. A task-aware candidate set at least ensures that a summarization request is choosing among models that are actually good at summarization, before cost or latency ever enters the decision.

Provider diversity for failover

The candidate set also has to account for what happens when a provider goes down. If every candidate for a task type happens to be hosted on the same provider, a routing policy that looks diversified on paper is a single point of failure in practice. Deliberately include at least one candidate from a second provider in every task-type group, even if it is not the first choice under normal conditions. That candidate rarely serves traffic, but it is what makes the fallback chain covered in Stage 4 possible instead of theoretical.

Security note: Provider diversity is a reliability property, not just a negotiating position. If a candidate set cannot survive one provider's outage without falling back to "return an error," it is not actually diversified — it is decorated.

3. Choose the routing signal

The signal comparison

The signal is the specific rule that turns a candidate list into one selected model for one request. Most production systems combine more than one signal, but it helps to reason about them separately before combining them, because they trade off differently and fail differently.

Decision rule: Never let a single signal override a quality floor. Cost-based routing should be allowed to pick the cheapest model that is expected to be good enough — it should never be allowed to pick a model that is expected to be inadequate just because it is cheap.

Cost-based routing: cheap first, escalate on low confidence

The most common pattern for cost-sensitive traffic is to send every request to a cheap, fast model first, then escalate to a stronger model only when the cheap model's own confidence signal is low. This works well when the confidence signal is trustworthy — a calibrated log-probability threshold, a self-consistency check across a few samples, or a lightweight classifier trained on cases the cheap model has historically gotten wrong. It works badly when "confidence" is approximated by something weak, like the cheap model's own stated certainty in its answer text, because models are frequently confident and wrong in exactly the cases this pattern is supposed to catch.

Reducing LLM costs without hurting quality covers the cost-reduction techniques this pattern depends on in more depth; the routing decision covered here is specifically about when to trust the cheap tier's own answer versus when to pay for a second opinion.

Latency-based routing

Latency-sensitive workflows — live chat, voice interfaces, anything with a user staring at a spinner — sometimes route by response time instead of, or in addition to, cost. This usually takes the form of racing two providers and taking whichever responds first, or maintaining a rolling p95 latency measurement per provider and routing away from the slower one. LLM latency optimization covers how to measure and reduce latency for a single call; the routing question here is different — it is about which provider to send the request to before you know how slow that provider will be right now. A rolling health score, refreshed on a short window, is more useful than a static assumption based on historical averages, because provider latency is not stationary — it degrades during regional incidents and improves after a provider ships a faster serving stack.

Task-based model selection in practice

Task-based routing depends entirely on the quality of the classifier that assigns a task type to an incoming request. A cheap keyword or regex classifier is fast and explainable but brittle — it is exactly the kind of classifier that produced the wrong-tier failure in this article's opening scenario. A small dedicated classification model is more robust to phrasing but adds its own latency and failure surface. Whichever approach you choose, treat the classifier itself as a monitored component: log its decisions, sample them for review, and track its accuracy the same way you would track the accuracy of the models it is routing to.

4. Design the fallback chain before you need it

What a fallback chain actually is

A fallback chain is the ordered list of what to try next when the selected model or provider fails to produce a usable response — a timeout, a 5xx error, a content-policy refusal, or output that fails validation. Operating LLM APIs in production covers how to classify a single call's failure; a fallback chain is what a routing layer does with that classification once it happens. The chain should be explicit and ordered, not "try something else" — for example: primary tier fails → retry once within budget → move to secondary provider → if that also fails, return a safe static response or queue for human review rather than serving a low-confidence answer silently.

Provider outage failover

A full provider outage is the fallback chain's hardest test, because it removes an entire tier of candidates at once rather than failing one request at a time. A well-designed chain degrades gracefully: requests that would have gone to the outed provider's primary model move to the secondary candidate for that task type, and the system's overall quality and cost both shift measurably — not silently. The ledger (Stage 5) is what turns that shift from a mystery into an explainable, temporary state.

A narrower version of the same problem is a "gray failure" — a provider that is technically reachable but returning degraded output, elevated error rates, or unusually slow responses without crossing a hard failure threshold. Health checks that only measure reachability miss this entirely. A rolling error-rate or latency-percentile health score, checked before routing rather than only after a request fails, catches gray failures earlier.

// Illustrative: a simplified routing function choosing between model tiers.
// Not tied to a specific vendor SDK — adapt the types to your gateway.

type Tier = "cheap" | "mid" | "premium";

interface RoutingDecision {
  tier: Tier;
  model: string;
  provider: string;
  reason: string;
}

interface ProviderHealth {
  available: boolean;
  rollingErrorRate: number;
}

function routeRequest(
  taskType: "code" | "support" | "summarize",
  confidenceFromCheapTier: number | null,
  health: Record<Tier, ProviderHealth>,
): RoutingDecision {
  // Stage 1 + 2 are assumed to have already narrowed taskType to an
  // eligible candidate set; this function only implements Stage 3 + 4.

  if (taskType === "code" && health.cheap.available && health.cheap.rollingErrorRate < 0.05) {
    return { tier: "cheap", model: "model-a", provider: "provider-1", reason: "task=code, cheap tier healthy" };
  }

  if (confidenceFromCheapTier !== null && confidenceFromCheapTier < 0.6) {
    if (health.mid.available && health.mid.rollingErrorRate < 0.05) {
      return { tier: "mid", model: "model-b", provider: "provider-2", reason: "low_confidence_escalation" };
    }
    return { tier: "premium", model: "model-c", provider: "provider-1", reason: "mid_tier_unavailable_escalate_premium" };
  }

  if (health.cheap.available) {
    return { tier: "cheap", model: "model-a", provider: "provider-1", reason: "default_cheap_first" };
  }

  return { tier: "mid", model: "model-b", provider: "provider-2", reason: "cheap_tier_unavailable_fallback" };
}

The function above returns a reason alongside the decision. That single field is the difference between a routing layer that can be debugged and one that can only be guessed at — it is the smallest useful version of Stage 5.

5. Worked example: the ticket that hit the wrong tier

Carry the opening scenario further. A support platform routes incoming tickets by a keyword classifier: tickets containing code-like tokens (backticks, function, a stack trace) go to a fast, cheap model tuned for syntax explanation. Everything else goes to a mid-tier conversational model. A customer submits a ticket that reads: "I was charged twice for my subscription — here's the error I got when I tried to fix my payment method: PaymentMethodError: card_declined."

The classifier sees the code-like token and routes the whole ticket to the cheap tier. The cheap model, which was tuned and evaluated on syntax questions, does exactly what it was built for: it explains what PaymentMethodError: card_declined means. It does not address the double charge, because nothing in its prompt or its evaluation history asked it to reason about billing. The customer receives a technically correct, contextually useless answer, and the ticket gets escalated to a human anyway — at which point the routing decision has cost more than if the ticket had gone straight to a human, because it also cost a model call and a round of customer frustration.

The root-cause fix is not to retrain or replace the cheap model — it was never the problem. The fix is to change what Stage 3 measures: replace or supplement the keyword classifier with one that considers intent alongside surface tokens, and add a rule that a ticket matching more than one task-type signal (here: both "code" and "billing") is treated as ambiguous and routed to the tier that can handle either, or flagged for confidence-based escalation rather than assigned confidently to the first matching keyword.

Field note: In practice, this exact failure pattern — a single strong keyword overriding a mixed-intent request — is one of the most common root causes teams find once they start reviewing routing decisions instead of only reviewing model outputs. It rarely shows up in a model-level evaluation, because the model being evaluated did nothing wrong.

6. Make the routing decision observable

What belongs in the ledger

Every step in the chain writes its own ledger row — a fallback that happens silently is a fallback you cannot audit later.

Stage 5 is what makes every earlier stage debuggable after the fact. At minimum, record per request: the eligible candidate set that was considered, the signal value that drove the decision (confidence score, latency measurement, task classification, or health status), the tier and specific model version that was selected, the reason string, the fallback path if one was used, and the resulting cost and latency. Store this alongside — not instead of — the trace of the model call itself, so a reviewer can move from "this response was wrong" to "this response was wrong because it was routed to tier A for this reason" in one step.

{
  "request_id": "req_9f21ac",
  "task_type_detected": "code",
  "task_type_signals": ["code_token_match"],
  "eligible_candidates": ["model-a:provider-1", "model-b:provider-2", "model-c:provider-1"],
  "selected_tier": "cheap",
  "selected_model": "model-a",
  "selected_provider": "provider-1",
  "routing_reason": "task=code, cheap tier healthy",
  "fallback_used": false,
  "cost_usd": 0.004,
  "latency_ms": 380
}

This is deliberately close to the alert and review-packet shape covered in AI agent monitoring — routing evidence should live in the same place as the rest of a request's trace, not in a separate system that only the gateway team can query.

Tracing a regression back to a routing change, not a model

The reason this matters in practice: a quality dashboard that only tracks "model X's success rate" cannot distinguish between two very different events — the model got worse, or the mix of requests being sent to that model changed. If a routing rule shifts a harder class of task onto a model that used to only see easy requests, that model's aggregate quality score will drop even though nothing about the model changed. Without a ledger, a team investigating that drop will look at the model, find nothing wrong, and either accept an unexplained regression or waste time swapping providers for a problem that was never about the provider.

LLM tracing explained covers how parent-run and child-span structure preserves this kind of context generally; the routing ledger is the specific set of fields a multi-model system needs on top of that structure to answer "why this model, for this request."

7. Compare the trade-offs

Cost, quality, latency, and lock-in

No routing policy optimizes all four of cost, quality, latency, and vendor independence simultaneously — every real policy is a stated trade-off among them, and the trade-off should be a decision someone made on purpose, not an accident of whichever signal was easiest to implement first.

A routing policy that never states its priority order will drift toward whichever dimension is easiest to measure — usually cost, because it is the simplest number to put on a dashboard, even when quality was supposed to be the actual constraint.

Vendor lock-in is a routing decision too

Multi-provider routing is sometimes framed purely as a reliability or cost play, but it is also a lock-in decision. A system that can only route within one provider's model family has effectively made a single-vendor commitment even if nobody signed a contract saying so — the switching cost is just deferred, not avoided. A candidate set that genuinely spans providers keeps that switching cost visible and manageable, at the price of the added integration and prompt-compatibility work every additional provider brings.

Decision rule: Treat "how many providers can serve this task type today" as a metric worth tracking on its own, separate from cost and latency. A candidate set that has quietly shrunk to one provider has lost its failover property even if nothing else about the routing policy changed.

8. Protect the security and failure boundaries

Provider outage mid-routing

A provider can fail partway through a request — after accepting it, before completing it, or with a partial stream already sent to the user. Decide in advance what happens to a partially streamed response when the underlying provider drops the connection: resuming from a different provider is rarely possible mid-stream, since context and partial output are provider-specific, so the practical options are usually to discard the partial output and restart on a fallback provider, or to surface an explicit "response interrupted" state rather than silently truncating. Neither choice is free — a discarded partial response wastes the tokens already generated, and a visible interruption is a worse user experience than a clean answer, but a silently truncated answer presented as complete is worse than either.

Secrets and API-key boundaries per provider

Each provider in a routing pool needs its own credential, and that credential should be scoped as narrowly as the provider's API allows — separate keys per environment, separate keys per team if the provider supports it, and rotation that does not require touching every routing rule at once. A gateway that centralizes routing is an attractive place to also centralize credential storage, which is correct, but it raises the stakes of a single leaked key: a compromised gateway credential does not just expose one integration, it can expose every workflow that routes through that provider. Isolate credentials per provider in the gateway's configuration so that revoking one does not require redeploying the whole routing layer, and keep provider credentials out of the routing ledger entirely — the ledger should record which provider served a request, never the credential used to reach it.

Silent quality regression from a routing bug

The most dangerous failure mode covered in this article is not a provider outage — outages are loud and get noticed quickly. It is a routing bug that changes which model serves a class of request without producing any error at all, the way the wrong-tier ticket example did. Nothing failed. No alert fired. The system worked exactly as configured; it was simply configured to make the wrong decision for a specific case. This is why Stage 5 is not optional: a quality regression with no accompanying error needs a ledger to even become visible, because there is no failure event to alert on — only a pattern in the ledger that a dashboard segmenting by routing_reason would surface, and a dashboard that only segments by model would not.

AI agent SLOs and error budgets covers how to track a quality indicator as a first-class signal alongside availability; a routing-aware system needs that quality indicator segmented by routing reason specifically, not just by model or by workflow, or this exact failure mode will keep hiding in the aggregate.

Before you ship: a routing release checklist

• [ ] Eligibility constraints are enforced as a hard gate, evaluated before any scoring or cost comparison.
• [ ] Every task-type candidate set includes at least one model from a second provider.
• [ ] The routing signal has a quality floor that cost or latency cannot override.
• [ ] The fallback chain is explicit, ordered, and ends in a safe default rather than a silent low-confidence answer.
• [ ] Every routing decision is recorded with tier, model, provider, reason, and outcome at request time.
• [ ] Quality dashboards can be segmented by routing reason, not only by model.
• [ ] Provider credentials are isolated per provider and excluded from the ledger itself.
• [ ] A recent quality regression has been checked against a routing-reason breakdown before being attributed to a model.

Decision rule: If the checklist above cannot be verified for a live routing policy, treat that as an open release blocker — a routing system that cannot explain its own decisions is not ready to carry production traffic, regardless of how well its models score individually.

Start with the smallest version of this framework that produces a real ledger: one task-type split, one cost- or confidence-based signal, one fallback step, and one recorded reason field. Expand the candidate set, the signal complexity, and the fallback depth only when a real investigation shows the smaller version was not enough — a routing policy earns its complexity through evidence, not through anticipation of problems that have not happened yet.

Frequently asked questions