Sixty percent. That is what your GPUs are doing while a very expensive training run crawls. Someone says it’s storage, it always sounds like storage, so the team buys the box the vendors push for AI, a WEKA or a VAST or a tuned Lustre, and the number sits at exactly sixty percent. The disks were never the bottleneck. The CPUs were pinned decoding JPEGs, and no filesystem decodes a JPEG.
That is why “does AI need its own filesystem?” is the wrong first question. A few of the biggest labs did build one, and what they changed is clever, we will get there. But for almost everyone the thing worth owning is not the storage. It is a straight answer to what your GPUs are stalling on, and that answer is usually cheap, rarely storage, and almost never a filesystem.
Where the old filesystem cracks
Every filesystem carries assumptions from the era it was built in, and AI training breaks three of them.
Files are big. A training set is not a few big files, it is tens of millions of tiny ones, images or audio clips of a few kilobytes each. Now the wall is not the disk, it is the metadata server: every sample is an open, a seek, and a close before you read a byte. The usual Linux mount for this, FUSE, tops out near 400,000 small reads a second, choked on lock contention in a single queue, a ceiling DeepSeek measured while building their own filesystem. At forty million files the queue is on fire long before the SSD notices.
And the obvious fix backfires. Cache metadata on the client and repeat lookups are free, except training opens the whole shuffled dataset at once, so the cache misses almost every time, each miss amplifies into more requests, and it squats on the RAM you needed for prefetch. FalconFS, which Huawei runs on ten-thousand-chip clusters, threw client-side metadata caching out entirely. Here it was worse than nothing.
The CPU sits in the middle. A normal read drags every byte through the CPU and the kernel. Fine on a laptop, fatal when you are pouring terabytes a second into a GPU. The fast systems cut the CPU out with RDMA, the GPU reading straight off the drive, and the most aggressive let the GPU issue the reads itself.
Access looks random but is not. A cache guesses what you will want next and is usually wrong. But training’s shuffle comes from a seed, so the order is fixed before the run starts, and one research system took brutal advantage of that.
What the GPUs are actually waiting on
The costliest mistake here is seeing an idle GPU and blaming the disk.
Two things stall a GPU, and they want opposite fixes. A fetch stall is waiting on bytes from storage; a faster disk cures it. A prep stall is waiting on the CPU to turn those bytes into something usable, decode, augment, batch into tensors, and a faster disk does nothing, because the disk is not what’s busy.
The Meta number everyone quotes as proof AI is storage-starved, half the GPU wasted, was a prep stall. The CPUs sat at ninety-two percent on feature work while the storage idled. Meta’s fix was disaggregated preprocessing, moving the CPU work onto its own fleet, not a filesystem. Faster storage would have bought them nothing.
The field’s crude test: pull the data loader and feed the model torch.rand. If the GPU jumps to a hundred percent, the pipeline is the problem. Then check the CPUs. If they are pinned, stop shopping for storage.
The fixes almost everyone needs
Once it is genuinely a fetch stall, the fixes run cheapest first, and most teams never reach the expensive end.
- Fix the layout. Tiny files are a layout problem, so change the layout: pack the millions of them into a handful of big sequential shards and the metadata storm evaporates, one open, one streaming read. WebDataset does it with plain tar; FFCV stores a tight binary format that sidesteps Python’s GIL, usually the real reason a loader can’t keep up; MosaicML streams shards from the cloud. Same move every time: turn a swarm of random reads into one sequential read, which every storage system is good at. This is the highest-leverage thing most teams can do, and DeepSeek’s alternative to doing it was a fifty-million-dollar cluster.
- Turn the loader up. Prefetch the next batch, add workers, pin memory. PyTorch ships all of it; teams just leave the dials low.
- Cache locally. Stage hot data on the node’s own NVMe, or drop JuiceFS or Alluxio in front of your object store, so the second epoch reads local flash, not the network.
The beautiful idea that mostly does not ship
There is one honestly new idea here, worth knowing even if you never run it. Training’s shuffle looks random but comes from a seed, so the exact order is known before the run. NoPFS took that literally: from the seed it computes the whole epoch’s access order up front and prefetches across the hierarchy, RAM to local SSD to a peer’s memory to the remote store, so each sample lands a beat early. Not a guess, a lookup. On spinning disks it cut training 5.4x.
The 5.4x came with fine print. It was measured on the slowest storage, spinning disks. On a larger dataset the gain fell to 2.4x, on a scientific one to 2.1x, and on fast flash it nearly disappears, because when there is barely any wait to begin with, a prefetcher has nothing left to hide. Later systems refined the idea: SHADE caches the samples that matter most to the model, and Quiver hands back a near-enough cached sample rather than stall on a miss. But that software rarely makes it into production. What survives is the lesson underneath: AI’s data access is predictable, not random, and that quietly shapes how every serious cache gets built.
When you actually need one
Some workloads outrun all of that, and a few labs built filesystems from scratch. What they changed beats “make it faster.”
DeepSeek’s 3FS starts by deleting a feature: no read cache. A read cache bets you will reread soon; training reshuffles every epoch, so it is dead weight. What is left is bare throughput: NVMe to GPU over RDMA, a zero-copy path past FUSE, metadata in FoundationDB with a one-hop file-to-location lookup, and CRAQ chain replication that stays consistent with no coordinator in the read path. Put 180 machines of that on InfiniBand with hundreds of clients and you get the headline 6.6 TB/s, on a cluster that costs tens of millions.
Intel’s DAOS drops the file tree entirely for an object store on persistent memory and NVMe, with POSIX bolted on for programs that still want it. BaM takes CPU-deletion to the limit, GPU threads issuing their own SSD reads. Meta rebuilt training storage on a flat object store with one-hop metadata, killing the layered lookups that cost a tenth of a second each, and tuned for tail latency over peak throughput, because a synchronous step moves at the speed of its slowest read: one straggler and every GPU waits.
Same three moves under all four: get the CPU out of the path, rebuild metadata for tiny files, pool the flash as one. Real, hard systems, built because past a certain scale storage is the wall. But that scale is hundreds of nodes and a team to run them, and if you are there, you already know.
Serving flips the whole problem
That is training. Serving flips it, and it catches people out, because they reach for the same heavy storage when it is the wrong tool. Inference does not want bandwidth, it wants to reuse the KV cache, the model’s running state, instead of recomputing it on every request. Hence Mooncake and LMCache, which park the KV cache in spare CPU memory, on SSD, or across the cluster. But the traces warn you: almost all the reuse is the prefix, the shared instructions at the front of every request, and the tail dies the moment a conversation ends. Alibaba’s traces suggest a cache on the GPU of about twice its own memory catches most of that reuse for business traffic, which means the elaborate storage fabric is often solving a problem the on-GPU cache already solved. That serving-side cache is a separate case study.
The honest answer
So, does AI need its own filesystem? A few labs do, and they have built clever ones. You almost certainly do not. The question worth your time is not which filesystem to buy, it is what your GPUs are stalling on. You find that out by profiling, an afternoon’s work, not by buying anything. CPUs pinned? Fix the preprocessing. Real I/O? Repack into shards, turn the loader up, cache to local flash, in that order, and measure after each. Reach for an exotic filesystem only after you hit a wall those cannot move, which for almost everyone is a wall they will never reach.
The fastest storage on earth cannot help a GPU waiting on a CPU. Knowing which one you are waiting on is the whole game. It is the same fight one layer up, in the storage lifecycle, and again in the choice of whether to buy the GPUs at all.