A top-end AI chip rents for three to four dollars an hour. At that price the one thing you cannot afford is for it to do nothing. Yet most of the time, that is exactly what it does.
The chip (a GPU, the specialized processor that runs AI models) can chew through data at terabytes per second. The storage it reads from hands data over at a few gigabytes per second, about a thousand times slower. So the GPU finishes its work and waits for the next batch to arrive. It is not slow. It is starving.
Everyone buys the chips. Almost nobody feeds them. Under every AI system runs a storage layer that decides whether the expensive hardware is ever actually used, and getting it wrong is the quietest, most expensive mistake in AI. As one storage engineer put it: your GPUs are idle, and that is not an AI problem, it is a math problem.
The wall
The mismatch is the whole story, and it fits in one line. A modern GPU wants data at terabytes per second; ordinary storage delivers single-digit gigabytes per second. That gap is a big part of why fleets costing millions can run well under full utilization.
Idle GPUs are pure waste. Drop a fleet from 90% busy to 60% and, on rented top-end chips, you are burning something like a dollar to a dollar-fifty per chip per hour for nothing. Across a thousand of them that is roughly a million dollars a month, spent on silicon waiting for a disk.
Six places a GPU waits on storage
Storage is not one thing under AI. It is a lifecycle, and the GPU can stall at every stage. Follow one workload through it.
- Feed. During training (teaching the model on your data), storage has to stream huge datasets into the GPUs fast enough to keep them busy. The data loader, the code that fetches and prepares each batch, is the single most common bottleneck. (A neat field trick: feed the model random numbers instead of real data. If it suddenly pegs at 100%, your storage, not your model, was the problem.)
- Checkpoint. Training runs for weeks, and hardware fails constantly, so the system periodically saves its state (a checkpoint) to disk so it can resume. For a 70-billion-parameter model that snapshot is about 140 gigabytes. Written the slow way it takes minutes, and every GPU in the job sits idle for the save. Written straight from the GPU to the drive it takes under a minute.
- Load. To serve a model you first copy its weights (the billions of numbers it learned) off disk into GPU memory. That is why a cold model can take 40 to 90 seconds before its first answer, much of it storage rather than compute. We went deep on that cold start in the serving-stack piece.
- Serve. While answering, the model keeps a running memory of the conversation (the KV cache) that spills from fast GPU memory down to slower tiers as it grows. Managing that spill is its own storage problem, a separate case study.
- Retrieve. If the model looks things up in your documents, it searches a store of embeddings (numeric fingerprints of meaning, also called vectors). At a billion documents those fingerprints cost too much to keep in memory, so they live on fast disk instead. Whether that retrieval is any good is a separate discipline.
- Offload. Finally, the cold stuff (old checkpoints, logs, rarely-touched data) moves to cheap bulk storage so the expensive fast storage stays free for hot work.
The flip: a conveyor belt, then a memory hierarchy
Here is the part almost everyone gets wrong, and it is the whole point. Storage design does not just vary by stage. It flips completely between training a model and serving one.
Training is a conveyor belt. It reads enormous datasets and writes enormous checkpoints, in big continuous streams. It wants raw bandwidth: hundreds of gigabytes per second, sustained, so the belt never stalls.
Serving is a memory hierarchy (layers of storage from small-and-fast down to big-and-slow). It loads weights, searches vectors, and shuffles the KV cache, all small, scattered reads, thousands of them at once from live users. It wants high IOPS and low latency: tiny requests answered in milliseconds, with even the slowest few held down.
Buy speed only where the wait is expensive
None of this means buy the most expensive storage everywhere. The rule is simpler: spend on fast storage only where it shortens an expensive wait.
Fast solid-state disk (NVMe) and direct chip-to-disk paths are worth it where a stall idles a room full of GPUs: a slow checkpoint, a slow cold start, a laggy vector search. Cheap, durable bulk storage (the kind that scales endlessly and costs little per terabyte) is right for everything cold: retained datasets, old checkpoints, logs.
And be honest about when it matters at all. If you are prototyping, running small jobs, or serving light traffic, plain cloud storage is fine and premium gear is money you do not need to spend yet. The threshold is not “AI sounds big.” It is a measurement: once you can see GPUs waiting on storage more than a few percent of the time, or cold starts blowing your deploy budget, or storage delay showing up in your slowest responses, that is when storage engineering starts paying for itself. Before that, do not.
The invisible floor
The GPU gets the headlines and the budget. The storage under it gets neither, right up until it becomes the reason the budget is wasted.
A GPU is only as fast as the data you can feed it. Buy the fastest chip in the world and starve it, and you have bought the world’s most expensive idle. The unglamorous work of feeding it, streaming, checkpointing, loading, retrieving, tiering, is the floor the whole thing stands on. It is invisible when it works, and it is exactly the kind of systems engineering we do.
If your GPUs look busier on the invoice than on the dashboard, the storage layer is where to look.