0:02So, I'm Andreas. I'm going to talk about automatic model routing. So, how it can save you money, how it can make the users happy, and how it is inherently flawed. So, first a little bit about myself. I'm Andreas. I was the co -founder and CTO at Replicate that was acquired by Cloud. almost exactly a year ago now and before that I was a machine learning engineer at Spotify. I was actually training models back then for music understanding and if you don't know what Replicate is, it is an inference platform for open source and proprietary language models, image models, video models, so a lot of the kind of thinking behind our automatic router came from the work I did there, but obviously the world has changed and now everyone has a model router and we need a model router as well.

1:00So I'm going to talk about how we built it, what the problem is trying to solve, and whether it's actually the right solution or whether there is something better we can do. So what is automatic model routing? So say you have an AI gateway like Cloudflare. AI Gateway or, you know, any of the other AI gateways that are out there. You have a bunch of models and users are, you know, using agents to talk to those models. Sometimes they ask simple things like publish these slides to Cloudflare pages. Sometimes they ask difficult things like solve the Navier Stokes problem. I wrote these slides a few weeks ago when Astra was still the best.

1:52model. For one of them, you want to use an expensive, capable model. And for some of them, you want to use a simple model. So the task is really like, what is the optimal way to let people solve their problems with as cheap models as possible, but not too cheap. So how does this work? So what most people do is they train a model, the small can - surrogate model that can run in the sort of the hot path of the agent to take a prompt or a trajectory or a subset of the trajectory and then output some probability distribution over the models. And that works, but it's really hard to update with new models.

2:42If you want to add a new model to that pool, you have to retrain the route. And that can take a lot of time, it can cause drift and all of these things. So what we did instead was, we made the router the trained model. Instead of predicting the model from the trajectory, we predict what task is this prompt trying We have a set of, I think it's 14 tasks that we're predicting over. a few dimensions. What's the complexity? What's the ambiguity? Is this a high -stakes task or just like a sort of prototype task? And how dependent is this query on previous context?

3:29Because different models handle that differently. So we train that model independently of the actual model selection. And then from there, we have this heuristic algorithm. I don't even call it the model because it's just a of a matrix of weights from the task probabilities and dimensions to the model probabilities. And that decoupling means that we can add a new model to the model pool without having to retrain the difficult to train router model.

4:05At the moment we only expose the actual router endpoint, but you can imagine actually exposing trained task probability, task prediction model as well, so people can make their own their own routers from this. So that's kind of another benefit of this decoupled architecture. It also makes it easier to reason about a model choice. So if the router says that this prompt should run on Opus, but we look at the prompt and it's not that complex. Like what happened there? We can debug it by looking at, is this the task predictor that failed or the model selector that failed? And most of the time it's the second, which means that we can tweak those weights without having to retrain the route.

4:55So that's our architecture.

5:01And how does it work? So how well does it work? When we released this a few weeks ago in the blog post, we saw 90 % Opus 5 .5 accuracy at 64 % of the cost. Now we have actually improved it further, and also with the new models that have come out, Haiku 5.5 and Sonnet 5.5, they form a kind of a smooth Pareto frontier. So, you know, it's like Anthropic owns the whole Pareto frontier at the moment, but that has also meant that we have even reduced... the cost even further. And so this is something that we are using internally at Cloudflare. And it's working. Some people have switched away from it. We don't sort of force people to use this, but most people who switch to it sort of stay on it and are okay with it.

5:56And it's, you know, it takes away some of the feeling that, like the cognitive debt you have when you know that you're spending a bunch of money, takes that away from me as the user and puts that on the platform. So that's kind of one of the goals of AI Gateway is that users shouldn't have to worry about their agents running off and spending a bunch of money. That should be a platform concern. So you can use this today. This is basically like a save money button. So you press the button and cost um you can just change your model slug on air gateway to cloudflare auto and it just works so that's great right um this problem solved not quite so there are some inherent ceilings in this type of model routing um the model routers they need to be really small so our model a modern BERT fine -tuned with some extra heads on it.

7:01It's 150 million parameters. It's really small. It runs in like 100 milliseconds on our edge GPUs. It's Cloudflare, which is great, but it's not very smart. Like, it's never going to be as smart as a real large language model. So the routing decisions or the task predictions it's making can, like, there's a very sort of hard ceiling on how good that can get. Another problem is that when you change to Haiku, you might get to the right answer, but it might take longer time. And that's a really hard thing to reason about. It's kind of like an agentic halting problem. How do you predict how many steps it's going to take to solve the problem? There are some benchmarks that measure this at the average, but that doesn't really generalize the old queries.

7:51And the third issue with this is that I use OPEN. differently to when I use Haiku like when I use Opus I can say there's a bug on about page and Opus will figure out what I mean and you can be very ambiguous and vague with Opus and just sort of understand my intent but with Haiku you have to be much more prescriptive and there's a bug on the back button that's 15 pixels too high and you need to fix that. That all sort of goes out the window when you don't know which model is gonna pick up the next problem. so that's another problem with with this type of automatic routing so what's the solution and I'm going to be talking about this is not like a Cloudflare product this is just kind of how what I've been doing since we shipped auto routing and I have in my local harness I have built sub -adient system that can offload tasks to cheaper sub-agents.

8:56So I think this is part of the solution. So, you know, a sub-agent tool, normally it has a prompt input and it has a description of what a sub-agent is. If you just add a model parameter to the sub -agent tool and you describe the models it has access to, how much they cost, what they're good for, and in the description to the sub-agent tool, you say something like use sub -agents to save money. Expensive or token expensive tasks that are not that hard should use a cheaper model and then if the orchestrator is something like Opus it reads those descriptions and it figures out that if I'm doing deep research on the internet if I'm reading log files stuff like that that goes to maybe Haiku, or maybe an even cheaper model in a sub-adient.

9:55And the thing that comes back to the context then is just the response. So these kind of sub-adients effectively become like a compaction of the individual sub-tasks. And that's been working really well. You can keep the orchestrator context really small. It doesn't grow very fast. All of those things happen in... sub-agents that can use cheaper models and that's obviously going to be a lot smarter than a tiny surrogate model. There's a bunch of other sort of harness level tricks here that people are exploring. Spotify released a blog post where they talked about their shunt system. I have no idea how to pronounce that. It's funny that the Swedish company makes a word that's unpronounceable for a Swedish person.

10:45So there the idea is that any large file reads at the file read tool level, you check is the output here larger than some threshold, and if it is, spawn a sub -adient to summarize the thing that the orchestrator wanted from that file read and put that into the context. There's a simpler version of that as well where large tool outputs again in... read file tool or in the shell exec tool you cap the output at you know say one kilobyte and if it's more than that you put it on a temporary file or on blob storage or something and then the agent uses grep and those kind of tools to read the file and compaction as well I don't know if any of you were at Ming's talk about AI gateway downstairs in the big orange globe earlier and she talked about a lot of things that that we do at Cloudflare on the EA Gateway level, like setting compaction rules so that we compact at 250,000 tokens instead of letting it go to 900,000 before we compact.

11:55Because obviously as the context grows, it's kind of exponential how much it costs to make each call. So there are a bunch of those kind of harness level tricks that people are doing. There are a lot more of them as well. I think... you're interested in this just take a lot some some of these kind of keywords feed them give them to your agent and let it sort of find the blogs and and and the the post where people talk about this it's sort of people come up with these new tricks all the time um so so summarize this um i think we should we should press the stop bleeding money button and use cloud for Auto. It does work, but I think we should also start experimenting with harness level routing to save money.

12:43Over the past like two years, I've built probably like half a dozen different harnesses for different things. It's really fun to build harnesses, especially when you have durable objects and you have frameworks like Flu and ADNs SDK and sandboxes on Cloudflare really tightly integrated. It makes it super easy to build these these harnesses and there's a lot of alpha in there like I think people think that building something like you know Claude code is really hard but like the Claude code code was leaked right and it's most of it was just like an agent loop and a bunch of tools and some some clever prompts there's not that much to it and you can find a lot of sort of things for your particular company that you can encode into tools and state machines stuff like that inside the harness so i encourage people to try out building their own harness and experimenting with these these cost saving techniques in there as well um yeah it's all i had there's not much to it it's it's uh just change the model slug and you're done yeah