Articles

Routing across models and providers

Choosing where a request goes: model and provider selection, router logic and its limits.

Reading: 4 minAutomation & AI

Article cover: Routing across models and providers

A request does not have to name the model that answers it. “LLM routing … offers an effective solution by first processing each user query through a router, which then determines the most suitable LLM to handle the query,” and the router “can direct simpler queries to smaller models and more complex ones to larger models, thereby balancing response quality with cost efficiency” (RouteLLM). Routing puts one decision between the caller and the model.

That decision — where a request goes — is this sheet’s subject, and a narrow one. What the request costs belongs to the cost-and-performance sheet; how many run at once to the concurrency sheet. A router is a component with its own inputs, failure modes and limits, and it makes two choices that are routinely conflated: which model, and which provider of that model.

Two decisions, one hop of indirection

A model decision selects between models of different size or family; a provider decision selects between deployments of a single model — the same weights behind different endpoints. LiteLLM comments the alias as “model alias -> loadbalance between models with same model_name” and lists “load-balance across multiple deployments (e.g. Azure/OpenAI)”. OpenRouter inverts the direction: the caller names a model and, by default, “requests are load balanced across the top providers to maximize uptime”. In both, the unit the router tracks is the deployment, not the model group — in LiteLLM, “cooldowns apply to individual deployments, not entire model groups”.

Choosing a model: a classifier that runs before the call

RouteLLM routes between two classes: “strong models, which provide high-quality responses at a high cost” and “weak models, which offer lower-quality responses at a reduced cost,” with the objective to “minimize costs while achieving a specific performance target” (line 24). Its mechanism trains a win-prediction classifier on preference data and compares the result to a cost threshold: “a higher value … enforces stricter cost constraints by favoring weak models more often, while a lower [value] biases toward higher-quality (but more expensive) strong models” (line 68).

The designs RouteLLM contrasts itself with — ensembling (LLM-Blender), cascading (Frugal-GPT), self-verification (AutoMix) — “rely on multiple LLM queries, which can increase latency,” whereas routing “routes each query to a single LLM” (line 29). A cascade pays for a call to discover it needed a bigger model; a router guesses first. The ideal is stated explicitly — quality from a single call per query, out-of-domain generalisation, and working “across a broad range of LLMs without retraining” (line 18).

Choosing a provider: the same model, served differently

Provider selection decides who serves an already-chosen model. OpenRouter’s default: it will “load balance requests across providers, prioritizing price,” prefer providers “that have not seen significant outages in the last 30 seconds,” and weight the stable ones “by inverse square of the price” (lines 180, 184–185). Setting sort to “price”, “throughput” or “latency” replaces that balancing with a fixed order, and “load balancing will be disabled” (lines 195–199).

Some provider fields rank candidates; others remove them. require_parameters means the request “won’t even be routed to that provider” when it lacks a parameter (line 1004); data_collection: "deny" admits only “providers which do not collect user data” (line 1064); zdr admits only endpoints that “have a Zero Data Retention policy” (line 1119). A filter that no provider satisfies leaves the candidate set empty: a routing rule with nowhere to route.

Provider choice is not cosmetic even for an identical model. The same hardware count from two clouds produced a “2x latency difference between 8xA100 servers from two cloud providers” (Databricks): the endpoint is part of the system, not a transparent pipe.

The limits of the decision

Routing does not remove a trade-off; it relocates it, and it is a guess made per request. RouteLLM’s own closing limitation is that “rather than there being a single best router for all queries, the decision of which router to use should be based holistically on latency and cost requirements” (line 480). The chosen strategy carries its own overhead too — LiteLLM recommends its simple-shuffle default for “best performance in production” while usage-based routing “adds significant latency due to Redis operations” (lines 133, 231); measuring that overhead belongs to the cost-and-performance sheet.

Two mechanisms are often mistaken for routing. Fallback after a failure is a separate concern (the fallback sheet): routing is the decision before the first attempt succeeds, fallback is what happens after it does not. And a deployment can be pinned to a live conversation, because deployments “do not share state, for example provider-side prompt caching, or … a conversation has to stay in one region” (line 653) — moving a session between providers can change its behaviour, not only its latency.

Level and prerequisites. L5 — the systems-engineering level: the reader must place a decision point outside the application, choose what it optimises for, and separate policy (a threshold, a filter) from measurement (overhead, quality). A router here is a component to be designed and measured, not a plugin to switch on. Prerequisites: the L1 and L2 sheets of this area, and the L3 and L4 sheets on tool calling and agent loops; the fallback, concurrency and cost sheets of this layer are its neighbouring halves.

Where to go next

References

  • Ong et al. — RouteLLM: Learning to Route LLMs with Preference Data — the strong/weak routing formulation, the cost threshold, the single-call-versus-cascade contrast, and the closing limitation.
  • LiteLLM — Router - Load Balancing — deployment-level routing: alias-to-deployment load balancing, the strategy list, per-deployment cooldowns, and session affinity.
  • OpenRouter — Provider Routing — same-model provider selection: the price-prioritised default, sort, and the policy filters.
  • Databricks (MosaicML) — LLM Inference Performance Engineering: Best Practices — the finding that the same hardware count on two clouds can differ in latency.
Nodrius Field Kit — cover

Get the Nodrius Field Kit

Enter your email and we'll send you the complete Field Kit: eight technical resources in one download.

Your email address is sent to Nodrius and used to deliver the Field Kit to you by email. Marketing messages are separate and optional: the checkbox does not affect your download, and you are only added to our updates list if you tick it. Privacy Policy.

Privacy & cookies

This site does not use cookies, analytics or tracking, and it does not profile you. There is nothing technical to switch off, so Accept and Reject change nothing about how the site works: either choice lets you browse everything normally.

The only difference is what your browser remembers: two entries in local storage — your choice here, so this notice is not shown again, and a marker that the Field Kit offer has already been made — so the offer is presented once and not on every page. They contain no identifier, they are not shared with anyone, and the site sets no cookie for either.

What we do with your data
How your email address is used when you ask for a Resource, how long it is kept and which rights you have — in the Privacy Policy.
What the site stores in your browser
Two entries in local storage — your choice on this notice and whether the Field Kit offer has been shown — and no cookies, no session storage, no third-party content: in the Cookie Policy.