A client called me about an AI bill that had gone from roughly $400 a month to just under $9,000, with no change in user numbers. Nobody had shipped anything unusual. Nobody could explain it.
It took an afternoon to find. A summarisation feature had been quietly moved from a nightly batch to a real-time trigger. Every record edit now fired a call. Users edit records repeatedly. The same document was being summarised eleven or twelve times a day, and the eleventh summary was identical to the first.
Nothing was broken. There was simply no mechanism anywhere in the system that could have noticed.
That is the state most teams are in with AI spend, and it is why FinOps has stopped being a cloud-cost topic and become an AI topic. Cloud costs at least grow predictably with traffic. Token costs grow with behaviour, and behaviour changes without a deploy.
Why This Is Harder Than Cloud Cost
Three properties make AI spend behave differently from anything else on your bill.
The unit cost is invisible at the call site. A developer writing await llm.complete(prompt) sees no price. The same line costs a hundredth of a cent or forty cents depending on the model and the context length, and nothing in the code tells them which.
Cost scales with context, not with requests. Ten thousand requests where the prompt grew from 800 to 12,000 tokens is a 15x bill increase with flat traffic. This is the single most common cause of a surprise, because context creep is invisible — a retrieval step returning more chunks, a conversation history that stopped being trimmed.
Usage is wildly uneven. On every AI product I have measured, a small fraction of users generate the majority of spend. Flat pricing averaged across users looks fine right up until the heavy tail arrives.
Attribution First, Always
You cannot manage a bill you cannot break down, and a provider dashboard showing one total per month is not a breakdown. Before any optimisation, log every call with enough tags to slice it:
await llmLog.record({
feature: "ticket_triage", // which product surface
userId: ctx.user.id, // who
tenantId: ctx.tenant.id, // which customer, for per-account margin
model: "small", // which tier
inputTokens: r.usage.input,
outputTokens: r.usage.output,
cachedTokens: r.usage.cached ?? 0,
latencyMs: Date.now() - started,
cacheHit: false,
costUsd: price(model, r.usage), // compute it now, not at reporting time
});
Computing the cost at call time rather than reconstructing it later matters more than it sounds. Provider pricing changes; a stored number is what you actually paid.
With that table you can answer the questions that drive every decision: which feature costs most, which customers are unprofitable, what your cost per active user is, and whether the average context length is drifting upward. That last one is your early warning system, and I now alert on it directly.
The Levers, Ranked by What They Actually Save
1. Route by difficulty. The biggest single lever, every time. Most teams send everything to their most capable model out of habit. Classification, routing, extraction, short rewrites and yes/no decisions run fine on a small model at a fraction of the price. An 80/20 split — small model for volume, large model for genuine reasoning — typically halves the bill with no user-visible change. Validate it against your eval set rather than assuming, but the answer is usually yes.
2. Cache, at two levels. Prompt caching from the provider for long stable prefixes, and your own cache keyed on a hash of the full input. The repeated-summary problem above was a cache miss that should never have been a call at all. On products with any repetition — and most have more than you would guess — a plain Redis lookup removes a double-digit percentage of traffic.
3. Cap output tokens. Output is the expensive half, often several times the input rate. Unconstrained models write four paragraphs where you render two sentences, and you paid for every discarded word. Set maxTokens deliberately on every call.
4. Shrink what goes in. Compact tool results before they enter the prompt, trim conversation history, retrieve five chunks instead of twenty. This usually improves quality as well, which makes it the least controversial change you can make.
5. Batch what is not interactive. Providers offer substantially cheaper asynchronous batch tiers. Nightly enrichment, bulk classification and backfills have no business running on the real-time endpoint.
6. Ask whether it needs a model at all. I have replaced two "AI features" with a lookup table and a regex this year. Both got faster, cheaper and more accurate. This is not a failure of AI, it is engineering.
Guardrails, Because Optimisation Decays
Every cost saving I have made has eventually been eaten by someone shipping a feature that did not know the rules. So the controls matter more than the one-off cleanup:
A hard per-tenant daily cap. Not an alert — a cap. When it trips, degrade to the small model or queue the work. This turns a runaway loop from a five-figure incident into a support ticket.
An alert on cost per request, not total spend. Total spend alerts fire after the damage. Average tokens per call drifting up 40% week over week catches it while it is still cheap.
Cost surfaced in pull requests. If a change alters the prompt, show the estimated cost delta from the eval run. Making the number visible at review time is worth more than a monthly report nobody opens.
A budget per feature, owned by a person. Unowned spend grows.
The Pricing Half
Cost control only gets you so far if the pricing model is structurally wrong. Flat-rate unlimited over a variable-cost service is a bet that your heaviest users stay light, and that bet loses eventually.
What has worked on products I have been involved with: a base subscription with a generous included allowance, expressed in a unit the customer understands — messages, documents, credits — and clear overage beyond it. Buyers accept it because it mirrors how they already buy usage-based services, and it means your revenue moves in the same direction as your cost.
The number to actually watch is gross margin per customer, computed from that logging table. On one product I worked on, the top 2% of accounts were being served at a loss, and nobody knew until we could slice the spend by tenant. Fixing that was a pricing conversation, not an engineering one — but it was only possible because the data existed.
That is really the whole discipline. Not clever savings, just knowing where the money goes before the bill tells you.
