Gracient.ai
Start a conversation
← Case studies
AI infrastructure & cost · Reference build · July 3, 2026

A serving stack that keeps GPUs busy without making users wait

A self-hosted large language model (LLM) is only cheap if its graphics cards stay busy, and only usable if they stay fast. This reference build is the serving stack that holds both at once: the GPUs never idle, and no user waits longer than you promised.

DomainAI infrastructure & cost
OutcomeGPUs run busy at a fixed latency SLO, so cost per token falls as load rises instead of climbing with it, with no change to the model.
Stack Serving engine (vLLM / SGLang)Continuous + chunked-prefill batchingFP8 + speculative decodingPrefill / decode disaggregationSLO-aware autoscaling + warm poolGoodput + cost-per-token metrics

Your GPUs are the most expensive thing in the building, and most of the time they are doing nothing. A GPU serving one request at a time can sit around 5% busy, waiting to move data, while it bills at the full rate. You are paying for silence.

You made the call to self-host the model. (If you are still weighing that call, see Before You Buy the GPUs.) The GPUs are yours now, and in the demo one request comes back in a second and everyone is happy. Then real traffic arrives, and the same fleet is somehow both slow and ruinously expensive, because the hardware is barely working.

Fast means every user gets an answer within the time you promised them, the target engineers call a service-level objective, or SLO. Cheap means the GPUs stay busy. Doing both at once is the whole job, and it is a systems problem, not a model one.

What we built

It is a single system, not a bag of separate tricks: a serving stack that sits between your users and the model and keeps the GPUs busy without making anyone wait too long. Three parts:

  • an engine that packs many requests onto each GPU,
  • a set of levers that push more work through without making users wait longer than promised, and
  • an autoscaler that adds and removes GPUs as traffic moves, so you never pay for idle ones.

Nothing here changes the model. It changes how hard the hardware works.

The serving stack Requests enter a gateway (the front door) and SLO-aware router, then a serving engine pool (vLLM or SGLang with continuous and chunked-prefill batching), then a band of optimization levers (shrink the model, guess ahead, split across GPUs, separate the phases, reuse past work). That feeds the GPU fleet. An autoscaler with a warm pool scales the fleet on queue depth and latency. The whole stack is judged on cost per million tokens at the latency SLO. The serving stack. requests gateway · SLO-aware routerthe front door: sends each request to the right pool serving engine poolvLLM / SGLang · continuous + chunked batching optimization levers · chosen by bottleneck shrink the model (FP8) · guess ahead (speculative decoding) · split across GPUs separate the phases (disaggregation) · reuse past work (KV cache) the GPU fleet autoscaler + warm poolscales the fleet on queue depth and latency; a warm pool hides cold starts IN LLM serving: the whole stack is judged on one number, cost per million tokens at your latency SLO.
One stack, one number. The engine packs requests onto the GPU, the levers push more through without breaking the SLO, and the autoscaler keeps the fleet sized to traffic. Everything is measured as cost per million tokens at a fixed speed target, your SLO.

The one chain that sets the bill

Everything traces to a single chain. Cost per token depends on how busy the GPU is (its utilization). How busy it is depends on the batch size, how many requests you pack onto it at once. And the batch size is capped by your latency budget, how long you will make a user wait.

Pack more requests onto a GPU and it works harder, so each token gets cheaper, by a lot: a busy GPU can be roughly 100x cheaper per token than an idle one. But the bigger the batch, the longer each user waits. So you cannot just crank it up. Throughput, the number of requests you serve per second, is the wrong target on its own, because throughput that makes everyone wait too long is useless to you. The number that matters is goodput: the requests per second that are still fast enough to keep your SLO, the speed you promised users. It is the throughput you can actually use, and it is the line the charts below are drawn against.

The chain that sets cost per token Your latency budget caps the batch size, batch size sets GPU utilization, and utilization sets cost per token. A small batch leaves the GPU near 5 percent busy and expensive per token; a big batch keeps it busy and cheap, but each user waits longer. One chain sets the bill. your latency budget batch size GPU utilization cost per token capssetssets small batch → the GPU idles near 5% busy → expensive per token big batch → the GPU runs busy → cheap per token, but each user waits longer IN LLM serving: a busy GPU can be about 100x cheaper per token than an idle one, so utilization is the whole game.
Idle silicon is the cost. You are not paying for intelligence; you are paying for a GPU whether it works or waits. The batch-size knob trades cost against latency, and the latency budget decides how far you can turn it.

Keeping the GPU busy

The engine is the foundation. vLLM and SGLang both pack requests together with continuous batching: instead of waiting for a whole batch to finish, the scheduler slips a new request in the instant an old one frees a slot, so the GPU never idles between requests. (Pick SGLang if your requests tend to share a long, identical opening, the same instructions or reference documents pasted in front of every question, and vLLM as the general default. The engine is the least interesting choice in the stack.)

The harder problem is that one big new prompt can stall everyone else’s token generation. Chunked prefill fixes it: the prompt is fed in small slices interleaved with ongoing work, so nobody stalls. It is now the default in both engines, and in the Sarathi research it raised serving capacity 2.6x on a single-GPU Mistral-7B and up to 5.6x on a much larger model spread across several GPUs, all within the latency budget.

Batch size is the master knob, and the whole art is finding the largest batch that still clears your SLO.

The goodput zone A chart of per-user latency against throughput. As you pack more requests onto the GPU, throughput rises but so does each user's latency. A horizontal line marks the latency SLO. To the left of where the curve crosses it, throughput meets the SLO and counts as goodput; to the right, extra throughput breaks the budget and is unusable. The throughput you can actually use. your latency SLO max goodput biggest batch under the SLO goodput ✓meets the SLO over the SLOtoo slow to use throughput (requests packed onto the GPU) → per-user latency ↑
Goodput, not tokens per second. Push throughput and per-user latency rises with it. The only throughput that counts is what stays under your SLO line; the sweet spot is the biggest batch that still clears it.

Pushing the frontier

Once the GPU is busy, you push more work through it, and the right lever depends on the bottleneck, not the hype:

  • Short on memory? Quantization. Store the model’s numbers in a smaller, coarser format. In FP8 (eight bits per number instead of the usual sixteen) that roughly halves the memory, and on the slow part of generating text roughly doubles throughput, for under 2% quality loss; it is the 2026 default. Drop to INT4 (four bits, smaller still but rougher) only when you must fit a bigger model on a smaller card.
  • Slow token by token? Speculative decoding. A small, fast draft model guesses the next few tokens and the big model checks them all in one pass; when the guess is right you get 2 to 3x the speed with identical output. It shines on code and structured text, less on open-ended writing.
  • Bigger than one GPU? Parallelism. Split the model across several GPUs, inside one machine over the fast links between cards (called NVLink), or across separate machines when it will not fit in one.
  • Phases fighting each other? Disaggregation. Reading the prompt and writing the answer stress a GPU differently and get in each other’s way when they share one. Splitting them onto separate pools of GPUs, now standard via NVIDIA Dynamo, served up to 7.4x more requests at the same speed target in the DistServe research.
  • Recomputing the same context? KV caching. As the model reads a prompt it builds a working memory of it; reuse that for a shared opening instead of rebuilding it every time. We cover that layer on its own in our caching case study; here it is one lever among many.
Which lever for which bottleneck A table matching symptoms to levers. Slow first token maps to chunked prefill and a prefill pool. Slow between tokens maps to a smaller batch, FP8, and disaggregation. Low throughput or high cost maps to a bigger batch, FP8, and more replicas. Out of GPU memory maps to INT4, shrinking and moving the KV cache, and parallelism. Bursty traffic maps to a warm pool and predictive scaling. Match the lever to the bottleneck. THE SYMPTOMREACH FOR slow first tokenchunked prefill · a dedicated prefill pool slow between tokenssmaller batch · FP8 · disaggregation low throughput, high costbigger batch · FP8 · more replicas out of GPU memoryINT4 · shrink and move the KV cache · parallelism bursty trafficwarm pool · predictive scaling
Diagnose, then treat. Raw tokens-per-second is a vanity number. You read the bottleneck from the metrics, reach for the one lever that fixes it, and re-measure as goodput. A lever aimed at the wrong bottleneck just adds cost.

Running it without burning money

None of this matters if the GPUs sit idle between traffic peaks. Scaling to zero saves money but hits the cold-start trap: spinning up a fresh GPU pod and loading a large model into memory can take minutes, during which the first user waits and the meter still runs.

So you keep a small warm pool always ready, and slice underused GPUs with MIG (multi-instance GPU) so one card serves several small jobs instead of one at 5%. The autoscaler watches queue depth and latency, not CPU, and adds capacity before the SLO breaks, not after.

When it is worth building

Honesty first: this stack only pays if the GPUs stay busy. Below roughly half-utilization, or with spiky, low-volume traffic, a hosted API is cheaper and simpler, the same conclusion as Before You Buy the GPUs.

Build this when your traffic is steady and large enough that the utilization you win covers the engineering, and when control over the model, the data, or the latency is worth owning.

Outcome

The result is measured, not promised. The GPUs run busy at a fixed latency SLO, so cost per token falls as load rises instead of climbing with it, and the number that matters, cost per million tokens at that SLO, sits on a dashboard rather than in a hope.

The model never changes. The hardware around it just stops idling.

This is a reference build. The stack, and the goodput tuning underneath it, is work we can do with you.