We were paying a frontier model to answer a yes-or-no question about six thousand records a day. Not reasoning. Not writing. Deciding whether a support message was a complaint or not.
Swapping that one call to a small model dropped its cost by well over 90% and made it noticeably faster. Accuracy on our eval set went from 94% to 93%. Nobody, internally or externally, ever noticed the difference.
That is the small-model argument in a sentence, and it is the least fashionable and most immediately profitable AI decision available to most teams right now. Everyone reaches for the biggest model by default. Most production work does not need it.
What "Small" Means Now
The label covers models roughly in the 1B to 15B parameter range — the compact tiers of the hosted families, and open-weight models like Llama, Qwen, Gemma, Phi and Mistral in their smaller sizes.
What changed is that these got good. Not equal to frontier models — they are not, and anyone claiming otherwise is selling something — but comfortably past the threshold for a large slice of real work. A small model today handles classification, extraction and short rewriting about as well as a flagship model did two years ago, at a fraction of the cost and latency.
The distinction that matters is not size, it is whether the task requires reasoning or recognition.
The Line I Draw
Tasks where small models consistently hold up:
Classification into a defined set. Intent, sentiment, category, priority, spam. This is the largest category of production AI by volume and it is where the savings are.
Structured extraction from messy text. Pulling fields out of an invoice, an email, a form. Give it a schema and it fills it.
Routing. Deciding which of six handlers or which of four tools. Cheap, fast, called constantly.
Short transformations. Rewriting to a tone, summarising a paragraph, normalising a string.
Filtering and pre-screening before an expensive step. Deciding whether something is worth sending to the big model at all.
Where they fall over, in my experience:
Multi-step reasoning. Anything where the answer depends on chaining three or four inferences. Small models produce confident, fluent, wrong chains.
Long context. Advertised windows are one thing; actually using information from the middle of 50,000 tokens is another, and small models degrade faster.
Agentic tool use with many tools. Choosing correctly among fifteen tools over twelve steps is exactly where capability shows.
Open-ended writing where quality is the product.
Route, Do Not Choose
The framing I push back on hardest is "which model should we use?" It assumes one answer for the whole application. Almost no application has one kind of task.
// One model per task, decided by the task — not one model per app.
const MODEL_FOR = {
classify_ticket: "small",
extract_fields: "small",
route_to_handler: "small",
draft_reply: "large",
analyse_incident: "large",
} as const;
export function complete(task: keyof typeof MODEL_FOR, input: string) {
return getProvider().complete({ tier: MODEL_FOR[task], prompt: input });
}
Once tasks are named, the mapping becomes a config decision you can revisit per task with your eval set, rather than an architectural argument. And when a small model gets better next quarter — they keep getting better — moving a task down a tier is a one-line change with a measurable before and after.
A useful pattern on top of this: escalate on low confidence. Run the small model, and if it reports uncertainty or fails schema validation, retry with the large one. You pay frontier prices only on the hard minority.
On-Device Is Now a Real Option
The other half of the small-model story is that these things now run on hardware people already own. Phones have neural accelerators. Laptops ship with dedicated silicon for this. A 3B model runs locally at usable speed.
The cases where that is genuinely the right call:
Privacy that must be structural. Health notes, legal documents, personal messages. "We don't store it" is a promise; "it never left the device" is an architecture. I built a browser game recently where the pose detection runs entirely client-side for exactly this reason — asking someone to point a webcam at their living room is only reasonable if nothing is uploaded.
Offline and unreliable connectivity. Field work, transit, poor coverage. A feature that stops working on a train is a feature people stop relying on.
Latency that has to be immediate. Autocomplete, live suggestions, anything reacting as someone types. A network round trip is a hundred milliseconds you cannot get back.
Per-request cost of zero. High-frequency, low-value inferences where a server bill would never be justified.
What people underestimate is everything around it. The model is a download of hundreds of megabytes to a few gigabytes — that is a real product decision, not an implementation detail. Device capability varies enormously, so you need a fallback path for older hardware. Quantisation to make it fit changes behaviour subtly, so evaluate the quantised build, not the original. And updating a model on devices you cannot reach is the same signed-artefact problem as firmware.
Fine-Tuning Makes More Sense Here
I usually argue against fine-tuning, because most teams propose it to solve a knowledge problem that retrieval solves better and cheaper.
Small models are the exception where it earns its place. Fine-tuning a compact model on a few thousand examples of one narrow task can lift it to frontier-level performance on that task specifically — and now you have a fast, cheap, self-hostable model that beats a general-purpose giant at the one thing you actually do.
The prerequisites are real, though: a genuinely narrow task, a few thousand good labelled examples, and an eval set to prove the gain. Without all three you are spending weeks to make something worse.
The Practical Move
Take your highest-volume AI call — the one that runs thousands of times a day. Run your eval set against a small model. Compare accuracy and cost.
If accuracy drops a point or two and cost drops by an order of magnitude, that is not a trade-off, that is a bug fix. In my experience, for classification and extraction work, that is the usual result.
The interesting question in AI engineering is no longer "how capable is the model." It is "what is the smallest model that passes my evals" — and most teams have never asked it.
