The AI Cost Teardown — Gracient.ai

Most of an AI bill is waste, not the price of the model (the AI itself). The number that shocks finance at month-end is the dozens of defaults left running: the same context (the background text you feed the model on every call) re-sent every time, reasoning (the model thinking out loud before it answers) left on for trivial questions, ten retrieved passages (snippets pulled from your own documents to ground the answer) where two would do, a rented GPU (the specialized chip that runs the model) billed around the clock.

So a cost engagement always starts the same way: a teardown. We follow one request end to end, find what is paying for nothing, and rank the fixes by risk and effort, lowest first. The map below is what we work from.

(This is a reference teardown, not a single client’s numbers. Every figure here is illustrative; on a real engagement we measure your own workload.)

Anatomy of an AI bill: 100 percent down to about 30 percent for the same answer, shown as the top-to-bottom path one request takes. The request enters a control layer that reuses repeated text and caps spending; a router sends easy work to a small cheap model and only hard work to the biggest, priciest model, about 60 percent off; the background text is assembled, with repeated parts reused for about 90 percent off and the rest trimmed and shortened; the model runs, with thinking-out-loud switched off, answer length capped, and off-peak jobs sent to a slower, cheaper queue for about 50 percent off; a final layer tracks cost per finished task and checks quality with an automated test. As a last resort, rewire: train a small model to copy the big one, run your own only if the chip stays about 70 percent busy, and shrink the search index behind your knowledge base. Figures illustrative. — **The teardown map.** Where each control sits in one request, top to bottom, and what it saves. Same answer, a fraction of the bill. Figures illustrative.

What we turn off first (lowest risk)

→ Caching. Long instructions and reference text get re-sent at full price on every call. Marked cacheable, the repeat costs about 90% less. The trap we check for: a timestamp or id at the top of the prompt (the text you send the model) that quietly breaks every cache hit.

→ Output caps. The model rambles to its default ceiling. We cap length and switch to structured output (a fixed format instead of free prose), so you stop paying for an essay nobody reads.

→ Reasoning. Newer models “think out loud” before easy answers, four to five times the tokens (the chunks of text you are billed for). We turn that down wherever an eval (an automated quality test) shows it changes nothing.

Then, the right tool for each job

→ Routing. Easy requests go to a small cheap model; only the hard ones reach the frontier model (the biggest, priciest AI). Commonly 60% or more, gated on an eval set so quality holds.

→ Send less. Semantic caching (reusing an answer when a new question means the same thing) for near-identical questions, trimmed retrieval (two passages, not ten), and prompt compression. Off-peak work moves to the batch lane (a slower, cheaper off-peak queue) at about half price.

→ Rewire, last. Only once the cheap wins are banked: distill (train a small model to copy the big one) a small specialist, self-host an open model (one you can freely download and run yourself, worthwhile only above ~70% GPU utilization), or shrink the vectors (the numeric search index behind a knowledge base).

What typically changes

The low-risk fixes alone tend to take a bill to roughly a third, before anyone rewires anything. The exact number depends on the workload, which is why the first deliverable of a teardown is your own measured baseline, not a slide.

Stacked, not magic. No single switch is a silver bullet; the cheap ones compound. Illustrative; we measure each workload.

How we work

This is careful, hands-on engineering: we study your stack in depth and sequence each change around live traffic, so the savings land without disruption. What we build lives in your own environment: caching, routing, batching, and a cost-per-task meter wired into your stack, so the savings live in the architecture rather than a one-off cleanup.

Want your own bill read line by line? Book a working session and we will run the teardown on your workload, whether or not we end up working together.

The thinking behind the method: The Case of the Runaway AI Bill, why cost sinks pilots before they ship, and Your AI Edge Is for Rent, what is worth owning versus renting.