A support email comes in: “I was double-charged for order 4471 and it still hasn’t shipped.” Your agent handles it end to end, and to do that it calls a language model about eleven times. It reads the message and decides what it’s about. It pulls out the order number. It judges the urgency. It picks a tool, fills in that tool’s arguments, and reads back the result. Then it does the same dance for the shipping question, and finally decides whether it can resolve the ticket or needs a human.
Count those eleven calls and something jumps out. Ten of them are clerical. Sort this into a category. Copy that number into a field. Pick one of four tools. None of them needs a genius, they need a competent clerk who is fast and cheap and never gets bored. Exactly one call, deciding whether a double charge is a real duplicate or a harmless authorization hold, actually needs judgment.
So here is the question almost nobody asks when they wire up an agent. If ten of the eleven steps are clerical, why is a frontier model, the most expensive and slowest option on the menu, doing all eleven? That is hiring a genius to file paperwork, eleven times, for every single request.
An agent is a pipeline, not a brain
The mistake is picturing an agent as one clever mind you drop a problem on. It isn’t. It’s a pipeline of small, typed decisions, most of them dull, a few of them hard. The useful question is not “which model should power my agent.” It’s “which model should power this step.”
Sort the steps and the pattern is stark. Classifying intent, extracting a field, choosing a tool, formatting that tool’s arguments, deciding whether to continue: these are bounded prediction problems with a short, structured answer. A small model, one you could run on a single commodity GPU, does them about as well as a frontier model. Only a handful of steps, genuine planning, open-ended synthesis, a judgment call on an ambiguous case, actually reward a bigger brain.
What a small model can actually do
“Small” here means a model in the rough range of one to eight billion parameters, the kind you can self-host on a single GPU or rent for pennies. The worry is that small means dumb. On these bounded steps, it doesn’t. Microsoft’s Phi-3-mini, at 3.8 billion parameters, scores 69% on the MMLU knowledge benchmark and 8.38 on MT-Bench, in the neighborhood of models many times its size. Tune a small model on your one task, or distill a big model’s behavior into it, and on that task the gap closes further.
The payoff is not subtle. An 8-billion-parameter open model like Llama 3.1 costs about $0.18 per million tokens; a mid-tier frontier model like GPT-4o runs $2.50 for a million in and $10 for a million out, and a top flagship several times more again. That is one to two orders of magnitude, per call, on the steps you make ten times a request, and the small model decodes faster because there is far less of it to run. Move ten of the eleven calls onto it and the bill for a request collapses, while the slow, expensive model is reserved for the one call that changes the outcome. The savings come from how many steps you move, not from finding a cheaper genius.
Two ways to escalate
“Escalate to a big model only when needed” hides two different mechanisms, and mixing them up is where designs get muddy.
The first is a router: a cheap check that looks at the request and decides, before any expensive call, whether it needs the big model. In our ticket, the double-charge judgment is the one step a router would flag for the frontier model. RouteLLM, an open framework from the LMSYS group, does this with a small learned model that scores how likely the strong model is to win on the prompt, then sends anything below a threshold to the weak one. On MT-Bench its best router held 95% of GPT-4’s quality at roughly a third of the cost, though the same routers save far less on harder benchmarks like MMLU, so treat the headline as a ceiling, not a promise.
The second is a cascade: try the small model first, and escalate only if its answer fails a check, low confidence, a schema that doesn’t validate, a tool that rejects the arguments. The shipping-status lookup in our ticket is a good candidate: let the small model try, and only reach for the big one if it stumbles. This is the cascade pattern FrugalGPT popularized, and in plain engineering terms it’s a fallback: run cheap, catch the failure, retry with something bigger.
Which to reach for? Often neither of the fancy versions. The simplest design, and the one most teams actually ship, is to route by step type at design time: classification always goes to the small model, planning always goes to the big one, no per-request decision at all. It has zero routing overhead and nothing to misfire. A learned router is itself a model call that costs tokens and latency and can route wrong (RouteLLM, for one, scores only the last message in the conversation), so a fixed rule usually wins until you have evidence you need more.
Can you trust it with the tool call?
The step engineers worry about most is tool use: if a small model formats a tool’s arguments as slightly broken JSON, the call fails and the pipeline derails. The fix is not a bigger model, it’s constrained decoding. Libraries like Outlines and llama.cpp take the tool’s schema and turn it into a hard filter on the model’s output: at each step they mask out every token that would break the format, so the model cannot emit invalid JSON. When our agent fills in the lookup_order tool’s arguments, a grammar guarantees it produces a valid {"order_id": ...} object every time. A 3B model under a grammar is a reliable form-filler.
With one honest caveat, and it matters. The filter guarantees the shape, not the values. A small model forced into a valid schema will always give you well-formed arguments, and can still put the wrong order number in the right field. Constrained decoding makes the tool call parseable, not correct, so you still validate what it produced.
Where small breaks
Small models earn the default, but they have a ceiling, and knowing where it is is the design. Two failures show up again and again.
The first is deep planning. Ask a small model to hold a multi-step plan in its head and it tends to lose the thread after a few hops, quietly dropping a constraint from three steps ago. The second is sharper and easy to miss: multi-turn tool use when a required piece is missing. Handed a request with no order number, a small model will often invent a plausible one and call the tool anyway, rather than recognizing it should stop and ask. It doesn’t fail loudly, it fails confidently, and in an agent one bad argument three steps back becomes a wrong answer at the end.
That is exactly the kind of step to escalate: not “everything hard,” but the specific moments that need judgment or the recognition that judgment is missing. Everything else stays small.
Right-size your agent
None of this is a call to abandon frontier models. It’s a call to stop paying frontier prices for clerical work. The move is boring and it works: trace one real run, label each model call by what it actually does, and move the bounded steps, the classifying and extracting and tool-formatting, onto a small model. Escalate only the steps that need judgment, and only after the small one signals it’s out of its depth. Then measure all three numbers that matter, quality, latency, and cost, and keep the change only if none of them slipped.
Do that and running your agent stops costing like a luxury. This is the seam we work in at Gracient: taking an agent that runs one expensive model eleven times and turning it into ten cheap, fast, reliable steps and one well-placed expensive one. A small model isn’t the budget version of your agent. On most of what an agent does, it’s simply the right tool, and the frontier model is the specialist you call in for the hard part. Let’s talk.