Model Routing Is Simple. Until It Isn’t.

IBM Research found that GPT-4.1 cost $155 for 417 AppWorld Test Challenge tasks, while Claude Sonnet 4.6 cost $79. The cheaper model on paper ended up nearly double the price in real-world testing. This gap existed because Sonnet’s lower cache-read pricing offset its higher base cost and longer reasoning steps. A router looking only at pricing sheets optimises against the wrong numbers.

Cost depends on usage patterns

Most routing systems treat model selection as a classification problem. Experience with agentic systems shows this is a systems optimisation issue. Three dimensions made the task difficult.

Actual cost depends on the interaction between the model, the workload, and the serving infrastructure. A router that only looks at pricing sheets is optimising against the wrong numbers.

Difficulty is invisible at routing time

A common strategy is to estimate task difficulty and send harder tasks to stronger models. This breaks down in two ways.

First, difficulty is often invisible at routing time. A request like “summarize this contract” looks simple, but might trigger retrieval, compliance checks, tool use, and multiple rounds of refinement before it is done. Meanwhile, a highly technical prompt might be handled efficiently by a smaller specialised model. You often do not know how hard a task actually is until execution is underway.

Second, even if you could perfectly estimate difficulty, it is only one signal among many. In production, routers need to balance cost, latency, model specialisation, and reliability simultaneously. Enterprise deployments pile on more: compliance requirements, data residency rules, privacy constraints, approved model lists. A task that would ideally go to one model might need to go elsewhere because of governance.

Routers are not solving one problem. They are constantly juggling cost, quality, latency, compliance, and reliability all at once.

Latency depends on serving conditions

It is tempting to think about latency purely in terms of model size. Bigger models are slower, smaller ones are faster. But what the user actually experiences depends on much more than that.

Routing itself adds overhead. Infrastructure factors — which hardware a model is running on, whether the cache is warm, how busy the endpoint is — often dominate end-to-end response times. A theoretically faster model can still produce a slower experience if the serving conditions are not right.

Then there is routing granularity. Routing once per task adds minimal overhead. But routing at every step — which gives you more flexibility to adapt mid-execution — means every additional decision point introduces latency and operational complexity.

A router that ignores the serving system is optimising against the wrong reality.

How IBM handled this

These lessons shaped how the team built their router. The key shift: they stopped treating routing as a classification problem and started treating it as an optimisation problem. Rather than asking “which model is best for this task?”, the algorithm optimises across cost, quality, and latency simultaneously. The system stays lightweight enough to avoid becoming a bottleneck itself.

The figure below shows the result on the AppWorld Test Challenge with a CodeAct agent. Each blue square is a different configuration of the router, tracing out a cost-accuracy frontier. The important thing is not any single point — it is that the router gives you a range of operating points to choose from depending on whether you want to prioritise cost, latency, or accuracy. Configuration 1 (latency-optimized) lands at 84% accuracy for $93 and 83s — a 21% cost reduction and 9% latency reduction compared to running Opus alone, with only a 4% accuracy drop. Configuration 2 pushes cost even lower.

Notice that a standard difficulty-based router (the teal diamond) lands in a similar accuracy range but at higher cost — it does not explore the full tradeoff space the way an optimisation-based approach can. And because the optimisation itself is lightweight (roughly 6 ms and 2 kB of memory per task), the router does not become the bottleneck warned about earlier.

What it means

The lesson is that routing is not really about choosing models. It is about optimising systems. Models are one variable — an important one, but just one among caching behaviour, infrastructure state, compliance constraints, and workload patterns.

When routing works well, it is rarely because it found the “best” model for a given task. It is because it found the best operating point for the entire system. That is a harder problem than classification, but it is the one worth solving.

IBM will share more about the technical details behind their approach in a follow-up post. In the meantime, if you are building routing into your own agentic systems, the team would like to hear what tradeoffs you are running into.