Your AI feature works. Usage is climbing, and last month the bill from your model provider crossed $8,000. In the budget review, someone asks the obvious question: “We’re renting intelligence by the token. Wouldn’t it be cheaper to buy our own GPUs and run the model ourselves?”
It sounds right. It is also the fastest way to either light money on fire or save a fortune, depending on a handful of numbers most teams never actually run. (A token, the unit you are billed for, is roughly three-quarters of a word.) So let’s run them.
The sticker price is the part that lies
When people picture self-hosting, they picture the GPU bill (GPUs, the specialized chips that run AI models): rent an H100 or two (a top-end Nvidia AI chip), run an open model (one you can download and run yourself), stop paying per token. That number is real, and it is the smallest part of the story.
The rented GPU is one line. The rest of the cost is quieter, and it is where self-hosting surprises people. You need engineers who can run the model in production (inference: the step where a finished AI model actually answers requests), the ones who get paged at 2am when a driver update breaks the cluster. You pay for idle time, because a rented GPU costs the same whether it serves zero tokens or four billion. You pay for power and cooling, since a single H100 draws around 700 watts before the room’s cooling overhead. And you pay to keep up: every time a better open model ships, someone has to re-tune and re-test the deployment. Add it up and the true cost of self-hosting runs roughly three to five times the GPU sticker.
So when does self-hosting win?
It comes down to volume. Because the model provider’s pay-per-use online service (its API) bills per token, its cost grows with every request. Self-hosting flips that: you pay a large fixed cost for the hardware, and each extra token is nearly free. Cross enough volume and the fixed cost wins.
The catch is that “enough volume” is not one number. Published break-even points span a thousandfold, from a few million tokens a month to well past ten billion. That is not because anyone is lying. It is because they are quietly comparing different things.
The first lever is what you are comparing against. Break even against a premium frontier model and the line arrives early, because that API is expensive. Compare against a rock-bottom open-weight endpoint (a ready-made online model you send requests to) that already sells tokens near cost, and you may never catch it. The second lever is the one almost every optimistic estimate ignores.
Utilization is the hinge
A rented GPU bills the same whether it is flat out or nearly idle. So the number that decides everything is its utilization: how busy you keep it.
Most real traffic is spiky: busy during business hours, near dead at night and on weekends. That means low average utilization, and low utilization wrecks the math, because you are paying for a full day of GPU while using a fraction of it. At around ten percent utilization, your effective cost per token can be roughly seven to nine times what it is on a well-fed GPU. This is why “we do 50 million tokens a day” does not settle the question. Fifty million streamed steadily is a cheap, busy GPU. The same fifty million in a few spiky hours is an expensive, mostly idle one. Our $8,000 team is the classic trap: plenty of monthly volume, but if it lands in weekday bursts, the GPUs they buy would sit idle most of the time.
The part the calculators skip: how you serve it
Two engineering choices move the cost as much as the buy-versus-rent decision, and neither shows up in a pricing table.
The first is the serving engine, the software that actually runs the model. Ollama is wonderful on a laptop and falls over under real concurrency (many users at once); vLLM and SGLang are the production standard, with vLLM pushing on the order of sixteen to twenty times Ollama’s throughput (work done per second) once traffic is heavy. Picking the wrong one can swing your real cost by 30 to 60 percent.
The second is quantization, which stores each of the model’s internal numbers in a coarser, smaller form (like rounding 3.14159 to 3.14) so it needs far less memory at close to the same quality for most workloads. Tools like Unsloth’s dynamic 4-bit can be the difference between needing four GPUs and needing one.
Get these two right and a self-host that looked like a loss becomes a win. Get them wrong and you pay cluster prices for laptop throughput.
The middle ground most teams miss
It was never a clean choice between “call the API” and “buy a rack of GPUs.” Serverless GPU platforms sit in between: some bill per token like an API but run open models (Fireworks, Together), others bill per GPU-hour but scale to zero when traffic stops (Modal, Baseten), so you skip both the idle cost and most of the ops burden (operations: keeping the servers running). And managed self-hosting runs open models inside your own network, for teams that need data to stay put, while someone else handles the 2am pages. The real decision has more than two doors.
The ground keeps moving
Even a perfect answer today has a short shelf life. API prices are falling fast, roughly an order of magnitude a year, sometimes more, for a fixed level of quality, and providers now cache repeated context at steep discounts on top of that. Every price cut pushes the self-host break-even further out. Pulling the other way, better open models and cheaper precision formats on new hardware keep making self-hosting more viable. The number you compute this quarter will move next quarter, which means this is a decision to revisit, not to make once and forget.
When it isn’t a cost question at all
Sometimes the spreadsheet does not get a vote. If your data is governed by HIPAA (US medical-data law), by GDPR data-residency rules (EU data law forcing data to stay in one country), or by a contract that says customer data cannot leave your perimeter, then a public API may simply not be allowed, and self-hosting or a private endpoint (a model running only on your own systems) becomes the answer regardless of price. In those cases the honest move is to accept the higher cost, or to keep the sensitive calls in-house and route the rest.
So, should you?
For most teams, most of the time, the API wins: your volume is low or spiky, you have no ops team to spare, and falling prices quietly do the saving for you. Self-hosting earns its keep when volume is both high and steady enough to keep the GPUs busy, when you have the engineering muscle to run it, or when compliance leaves no choice. And a large middle ground is best served by a hybrid: a self-hosted baseline kept full, bursting to an API for spikes, with sensitive traffic kept at home.
The uncomfortable truth is that the answer is “it depends,” and the “it” is a dozen numbers about your traffic, your models, and your rules that no blog post can know for you. That is the work we do with teams at Gracient: run the real total cost on your actual traffic, choose the serving stack and quantization that make the winning option genuinely win, and build the routing layer so you are never stuck on the wrong side of a line that keeps moving. If your AI bill just made someone ask “shouldn’t we self-host?”, the answer starts with the math. Let’s talk.