Routing across models and providers
Choosing where a request goes: model and provider selection, router logic and its limits.

A request does not have to name the model that answers it. “LLM routing … offers an effective solution by first processing each user query through a router, which then determines the most suitable LLM to handle the query,” and the router “can direct simpler queries to smaller models and more complex ones to larger models, thereby balancing response quality with cost efficiency” (RouteLLM). Routing puts one decision between the caller and the model.
That decision — where a request goes — is this sheet’s subject, and a narrow one. What the request costs belongs to the cost-and-performance sheet; how many run at once to the concurrency sheet. A router is a component with its own inputs, failure modes and limits, and it makes two choices that are routinely conflated: which model, and which provider of that model.
Two decisions, one hop of indirection
A model decision selects between models of different size or family; a provider decision selects between
deployments of a single model — the same weights behind different endpoints. LiteLLM comments the alias as
“model alias -> loadbalance between models with same
model_name” and lists “load-balance across multiple deployments (e.g. Azure/OpenAI)”. OpenRouter inverts
the direction: the caller names a model and, by default, “requests are load balanced across the top
providers to maximize uptime”. In both, the unit the router tracks is the deployment, not the model group
— in LiteLLM, “cooldowns apply to individual deployments, not entire model groups”.
Choosing a model: a classifier that runs before the call
RouteLLM routes between two classes: “strong models, which provide high-quality responses at a high cost” and “weak models, which offer lower-quality responses at a reduced cost,” with the objective to “minimize costs while achieving a specific performance target” (line 24). Its mechanism trains a win-prediction classifier on preference data and compares the result to a cost threshold: “a higher value … enforces stricter cost constraints by favoring weak models more often, while a lower [value] biases toward higher-quality (but more expensive) strong models” (line 68).
The designs RouteLLM contrasts itself with — ensembling (LLM-Blender), cascading (Frugal-GPT), self-verification (AutoMix) — “rely on multiple LLM queries, which can increase latency,” whereas routing “routes each query to a single LLM” (line 29). A cascade pays for a call to discover it needed a bigger model; a router guesses first. The ideal is stated explicitly — quality from a single call per query, out-of-domain generalisation, and working “across a broad range of LLMs without retraining” (line 18).
Choosing a provider: the same model, served differently
Provider selection decides who serves an already-chosen model. OpenRouter’s default: it will “load balance requests across providers, prioritizing price,” prefer providers “that have not seen significant outages in the last 30 seconds,” and weight the stable ones “by inverse square of the price” (lines 180, 184–185). Setting sort to “price”, “throughput” or “latency” replaces that balancing with a fixed order, and “load balancing will be disabled” (lines 195–199).
Some provider fields rank candidates; others remove them. require_parameters means the request “won’t even
be routed to that provider” when it lacks a parameter (line 1004); data_collection: "deny" admits only
“providers which do not collect user data” (line 1064); zdr admits only endpoints that “have a Zero Data
Retention policy” (line 1119). A filter that no provider satisfies leaves the candidate set empty: a
routing rule with nowhere to route.
Provider choice is not cosmetic even for an identical model. The same hardware count from two clouds produced a “2x latency difference between 8xA100 servers from two cloud providers” (Databricks): the endpoint is part of the system, not a transparent pipe.
The limits of the decision
Routing does not remove a trade-off; it relocates it, and it is a guess made per request. RouteLLM’s own closing limitation is that “rather than there being a single best router for all queries, the decision of which router to use should be based holistically on latency and cost requirements” (line 480). The chosen strategy carries its own overhead too — LiteLLM recommends its simple-shuffle default for “best performance in production” while usage-based routing “adds significant latency due to Redis operations” (lines 133, 231); measuring that overhead belongs to the cost-and-performance sheet.
Two mechanisms are often mistaken for routing. Fallback after a failure is a separate concern (the fallback sheet): routing is the decision before the first attempt succeeds, fallback is what happens after it does not. And a deployment can be pinned to a live conversation, because deployments “do not share state, for example provider-side prompt caching, or … a conversation has to stay in one region” (line 653) — moving a session between providers can change its behaviour, not only its latency.
Level and prerequisites. L5 — the systems-engineering level: the reader must place a decision point outside the application, choose what it optimises for, and separate policy (a threshold, a filter) from measurement (overhead, quality). A router here is a component to be designed and measured, not a plugin to switch on. Prerequisites: the L1 and L2 sheets of this area, and the L3 and L4 sheets on tool calling and agent loops; the fallback, concurrency and cost sheets of this layer are its neighbouring halves.
Where to go next
- Automation & AI — the macro-area this sheet belongs to.
- Applications — the node this sheet sits in.
References
- Ong et al. — RouteLLM: Learning to Route LLMs with Preference Data — the strong/weak routing formulation, the cost threshold, the single-call-versus-cascade contrast, and the closing limitation.
- LiteLLM — Router - Load Balancing — deployment-level routing: alias-to-deployment load balancing, the strategy list, per-deployment cooldowns, and session affinity.
- OpenRouter — Provider Routing — same-model provider selection: the price-prioritised default, sort, and the policy filters.
- Databricks (MosaicML) — LLM Inference Performance Engineering: Best Practices — the finding that the same hardware count on two clouds can differ in latency.