Stay up to date with AI tools, model releases, and developer workflows that matter.

In a recent development, weekly. Free. One click to leave.

7 Day Free Trial. Cancel Anytime.

In a recent development, "Do you know the closest ATM?" and "I was charged twice, the refund never arrived, and now my card is blocked" are both customer messages. Most AI apps send both to the same large model and pay large-model prices for each one.

Model routing fixes that. A router looks at each request and sends it to the cheapest model that can handle it, and keeps the expensive model for the requests that need it.

In this guide we'll cover what model routing is, how routers decide, and what it saves. Then we'll build a small router in about 60 lines of JavaScript and run it over 200 customer messages. At one setting it matched a large model's accuracy for about a third of the cost. At another it came within a point and a half for 89% less.

It has been revealed that model routing in AI is the practice of sending each request to the model best suited to it, instead of sending every request to one model. A router sits between your app and a pool of models, reads each request, and picks a model based on what you care about: cost, speed, or quality.

You'll also see it called AI model routing, LLM routing, or LLM model routing. They all mean the same thing. The pool might be a small and a large model from one provider, or models from several providers behind one API.

According to recent reports, the idea rests on one observation: most requests are easy. A small model answers them as well as a large one, in a fraction of the time and for a fraction of the price. The router's whole job is finding the hard ones.

The diagram below demonstrates a typical AI model routing architecture.

It has been revealed that model routing works in three steps: the router reads the request, decides which model should answer it, and forwards the request to that model. The middle step is where routers differ. Routing tools decide which model to use in one of four ways.

Rule-based routing uses rules you write by hand. Requests from free-initiative customers go to a small model. Anything tagged as code goes to a coding model. Prompts over a certain length go to a model with a bigger context window. It's simple, predictable, and free to run, but the rules only know what you thought of in advance.

Intelligent model routing uses a model to make the decision. A small, fast classifier reads each request and predicts which model in the pool will answer it well. You'll also see it called smart or dynamic model routing.

Industry observers note that two more terms come up here. Task-aware routing picks a model by the kind of job, like summarising, coding, or answering a support question. Context-aware routing also looks at what surrounds the request, like the conversation so far, the user, or the tools available.

The major platforms work this way. Microsoft describes its Foundry model router as "a trained language model" that picks a model for each prompt in real time. Google indicates Gemini Notebook uses its latest Decisions API as a real-time router, serving everyday queries with up to 40% lower latency. I've tested the Google Decisions API against Jev on the same messages we'll use below.

It has been revealed that cascade routing tries the cheapest model first and only moves up when the answer isn't good enough. The usual test is confidence: if the cheap model is sure, keep its answer. If it isn't, send the request to a stronger model. It's also called confidence-based routing, and it's what we'll build below.

A cascade needs no separate classifier, because the first model's own confidence does the routing. The trade-off is that escalated requests pay for two models and wait for both.

In a recent development, fallback routing is about staying up, not saving money. If a model times out, hits a rate limit, or its provider has an outage, the router sends the request to the next model on a list. Most production routers combine fallbacks with one of the three methods above.

The main benefits of model routing in AI systems are lower cost, faster responses, and fewer outages, without giving up quality on the hard requests.

We'll build a cascade router for a job every support tool has: sorting customer messages. Each message gets one of ten labels, like atm support or request refund.

It has been revealed that the test set is 200 messages from Banking77, a public set of banking support messages: ten labels, 20 messages each. They're the same 200 messages from my zero shot classification test, which is where the single-model numbers below come from.

Put both keys in a .env file and run the examples with node --env-file=.env. It's the same setup as in my guide to getting a YouTube API key.

It has been revealed that create router.mjs. The first function sends the message to Jev with one multiple-choice question, and returns the label Jev picked along with its confidence:.

It has been revealed that confidence runs from 0 to 1. When nearly all the probability sits on one label, it's close to 1. When Jev is torn between two labels, it drops.

In a recent development, the second function asks Claude Opus 5.5 the same question. The output_config block limits the answer to one of our labels, so there's no free text to parse. SitePoint's guide to LLM structured output with JSON Schema explains how that works.

The router itself is about a dozen lines:.

If Jev is at least 80% confident, its answer goes straight back. If not, Claude answers instead. And if Claude fails, the router falls back to Jev's answer, because a shaky answer beats an error page.

Try it on an easy message and a harder one:.

Industry observers note that jev was sure about the lost card. On the ATM question it leaned towards card acceptance with a confidence of 0.64, so the router asked Claude, which got it right.

Because cheap, strong, and threshold are options, the same function routes between any two models.

It has been revealed that model routing reduces AI costs by keeping easy requests on a cheap model and paying for the expensive one only when it's needed. Here's what that looked like on our 200 messages, starting with each model on its own:.

Opus gets six more messages right out of 200, and costs more than 120 times as much.

Now the router, at four confidence thresholds:.

Sources indicate that speed improved too. At the 0.8 threshold, the messages Jev kept came back in a median of 227 ms. The ones that escalated took about 2.1 seconds, because they waited for both models.

A note on method: I ran every message through both models once and saved the answers, then replayed the router over them at each threshold. That's how you compare thresholds without paying for the same test four times.

Then I ran the router live to check. At 0.95 it scored the same 95.0% and sent 61 messages to Opus. At 0.8 it scored 93.5% and sent 20, a point under the replay. Two answers moved: one of Jev's confidence scores landed at 0.82 instead of 0.78, just over the line, and Opus changed its answer on one message. Scores near the threshold shift a little between runs, so leave yourself a margin.

According to recent reports, the choice of cheap model matters less than you'd think. With Google's Gemini Decision 1 in first place instead of Jev, a 0.7 threshold reached the same 95% for US$0.28 per 1,000 messages.

Model routing can cost a little quality, and the threshold decides how much. In our test the gap to Opus ranged from nothing at 0.95 to two points at 0.7.

In a recent development, the loss comes from one place: answers the cheap model got wrong while feeling sure. At the 0.8 threshold, Jev kept five wrong answers because its confidence was above the line. Opus got four of those five wrong as well, so only one was a real loss: "Do all ATMs take this card?", which Jev filed under card acceptance with a confidence of 0.91.

Escalating isn't risk-free either. On one message, Jev had the right label with a confidence of 0.44, and Opus overruled it with the wrong one.

Industry observers note that to optimize costs using AI model routing, measure first, route second, and keep measuring. These six steps cover LLM model routing cost optimization for any pair of models:.

In a recent development, the return is easy to estimate once you know the split. If nine in ten requests stay on a model that costs 1% as much, the bill drops by about 89%, which is what we measured.

In a recent development, several inference providers have multi-model routing built in, and most AI gateway platforms support model routing too, so you don't have to write your own. As of November 2026, these are the top model routing solutions for AI apps. Each name links to the tool's model routing docs:.

Hosted model routing platforms for large language models decide for you, which is the fastest way to start. A hand-built router like ours takes more work, but you choose the signal and you can see why every request went where it did.

An LLM router is the component that does model routing for large language models. It can be a function in your own code, like the one in this guide, a hosted model such as Microsoft Foundry's model router, or a feature of an AI gateway.

Industry observers note that smart model routing is another name for intelligent model routing: a model, not a hand-written rule, decides where each request goes. Dynamic model routing means the same thing.

According to recent reports, the ROI of AI model routing depends on how much of your traffic a cheaper model can handle. In our test, a 0.8 threshold kept 89.5% of messages on the cheap model and cut the cost per 1,000 messages from US$1.96 to US$0.22, for about a point of accuracy. The build was about 60 lines of code.

In a recent development, openRouter's auto model routing works through its Auto Router. You set the model to openrouter/auto, and a decision model picks a model and a reasoning effort for each request. You pay the standard rate of whichever model it selects, with no extra fee.

OpenClaw model routing is a primary model plus an ordered list of fallbacks. In the OpenClaw model routing configuration you set agents.defaults.model.primary and agents.defaults.model.fallbacks, and an agent can override the fallback list. When a model fails, OpenClaw first rotates auth profiles on the same provider, then moves to the next model on the list, as its model failover docs describe. That makes it fallback routing, not cost-based routing.

An AI gateway simplifies model routing by putting every model behind one API, one set of keys, and one place for logs. Your app calls the gateway, and the routing rules live in config instead of code, so changing a model doesn't need a deploy.

Sources indicate that mid-stream failover is when a router switches models after a streamed response has already started, because the first model failed partway through. It's the hardest kind of failover, because the user has already seen part of an answer. Many routers only fail over before the first token arrives and return an error after that, so check what your gateway does.

7 Day Free Trial. Cancel Anytime.

β†’ Read original article