A support assistant looks cheap in the demo. One question, one answer, a few cents.
Then real traffic arrives, and the same forty-page policy doc, the same system prompt (the fixed instructions that set the assistant’s role), and the same tool definitions (the list of outside actions the model may call) ride along on every call. The model reprocesses all of it, every time. Cost and the slow-request latency climb with volume, not with value. You are not paying for intelligence. You are paying to make the model re-read the same pages, all day.
A smaller model loses quality. Fine-tuning, retraining the model on your own data, is slow and risky. The cheap, safe fix is to stop recomputing what has not changed. That is caching, and done well it is a systems problem, not a setting.
What we built
One layer, sitting in front of the model, with three jobs:
- answer a repeat question without calling the model at all,
- reuse the static context so the model never re-reads it, and
- stay correct while doing both.
Nothing here touches the model. It changes what surrounds it. The rest of this is those three jobs and the decisions inside them.
Two terms run through all three jobs, so here they are up front. A token is a word-piece, the unit a model reads and writes one at a time. A model’s KV cache is the working memory it builds as it reads a prompt, its saved notes on the words already seen, so it does not start over on each token. The shared opening chunk of that memory is the prefix, and reusing it is how a later request skips straight past what it has already read.
Skipping the model
The top of the layer is the biggest win. A semantic cache answers a question without calling the model at all: embed the new question (turn it into numbers that capture its meaning), find the closest one already answered, and if they are close enough, return the stored answer. On a support line where the same questions arrive all day, that skips a large share of calls, in milliseconds, billing nothing.
Under the hood it is a small pipeline. The question becomes a vector (that list of numbers), a per-tenant store returns the nearest earlier question, guards decide whether that match is safe to reuse, and a miss falls through to the model, whose answer is written back for next time.
The risk hides in “close enough.” Similarity is a threshold, and the threshold trades savings against safety. In one AWS test, loosening it from 0.99 to 0.90 lifted the hit rate, the share of questions answered straight from the cache, from 24% to 75%, with accuracy still near 92%. Go lower and wrong answers slip through.
A wrong answer here is not an error you can see. It is a confident, wrong reply returned with a 200 OK, the same success code a correct answer carries, invisible on every dashboard. The worst case crosses users: someone asks “where is my order?” and gets another customer’s status, because the two questions look nearly identical to the model. A miss costs one extra call. A false hit costs the customer.
Keeping it honest
So the semantic cache never answers on similarity alone. A stored answer is served only if it clears a short gate, and anything that fails just falls through to the model:
- Same customer. The cache is namespaced per tenant, so an answer can never cross people. That alone closes the “where is my order” leak.
- Same versions. The prompt, model, and embedding version are part of the key (the label an entry is stored and looked up under). Change any one and old entries stop matching, because they are no longer comparable.
- Still fresh. A time limit by data type, plus a fingerprint of the source documents, so a changed policy retires the answers built on it.
- Exact entities. Names, IDs, dates, and amounts must match exactly, not just in meaning, so “refund order 4471” never answers “refund order 4472.”
- Route threshold. Set high where a mistake is expensive, looser on plain FAQs, with a quick verifier on the sensitive routes.
None of this shows in a demo. All of it is why the cache is safe to leave on in production.
Reusing the context
The second job is the bigger, quieter saving: never make the model re-read the static context.
When a model reads a prompt it builds that KV cache, its working state, so it does not start over on each token. Keep that state and a request that begins the same way jumps straight to answering. Who keeps it is the only real fork.
If you call a hosted API, the provider keeps it for you. Their “prompt cache” is that same KV cache. You cannot manage it, but you can drive it: put the stable content first, mark the cache point, and keep the changing parts last, or a single new character near the top throws the whole prefix away. Done right, cached reads bill at about a tenth of a fresh one. You pay per read, metered, and writes cost a premium, so cache only what actually repeats.
If you run the model yourself, the KV cache is yours, and it turns into a storage problem. The state is huge and GPU memory is the scarcest thing in the cluster, so it spills across tiers like any storage cache: the hottest blocks (chunks of that cached state) in GPU memory, warm ones in CPU memory, cold-but-costly ones on SSD (local disk), pooled across machines by projects like LMCache and Mooncake. For our assistant, the forty-page policy doc is read once and stays warm, not reprocessed on every ticket.
Which block stays where is an eviction decision, choosing what to drop to make room, and the naive policies waste money. In one large provider’s traces, a tenth of the blocks carried three-quarters of all reuse, and a workload-aware policy beat plain LRU (throw out the least-recently-used block) and cut mean response time by up to 40%. Keeping the right prefix hot is the whole game.
Put together, the self-hosted cache is a small stack, and every layer of it is yours to tune:
What we don’t cache
A caching layer is defined as much by what it refuses:
- Never across customers. Convenience is not worth a leak.
- Never volatile data. Prices, balances, “the latest” anything, these skip the cache by default. A fast stale answer is worse than a slow fresh one.
- Never the whole context blindly. Caching everything can add latency instead of removing it, so each route earns its cache or does without.
- Never an unmeasured saving. Every layer reports what it saved, so “90% off the cached tokens” is never mistaken for “90% off the bill.” They are different numbers.
The parts are commodity, the build is not
Every box in these diagrams is something you can download: an embedding model, a vector store, vLLM, LMCache. Wire them together and you get a cache that is fast and unsafe.
What that does not give you is a cache that never leaks across customers, never serves last week’s policy, and keeps the right prefixes hot under real load. That part is not a library. It is judgment applied per route: which tokens must match exactly, where each threshold sits, how an entry is keyed and invalidated, which blocks are worth GPU memory, and the measurement loop that proves all of it on real traffic.
The cache never understands your domain, and it does not need to. You understand it, and you encode that understanding in the guards and the eviction policy. That encoding is the work.
What changes
The layer is measured, not promised. The static context is computed once and reused, so cost and latency follow value instead of volume. The cache that could answer wrong is boxed in by the gate and watched by a false-positive rate that catches drift before a customer does. And whether the KV cache is yours or the provider’s, it is the same machinery: engineered end to end when you run it, driven correctly when they do.
The model never changes. What surrounds it stops paying twice.
This is a reference build. The layer, and the eviction and invalidation underneath it, is work we can do with you.