Gracient.ai
Start a conversation
← Case studies
Data & Retrieval · Reference build · July 9, 2026

Your RAG Is Recommending an API You Deleted

A RAG freshness build. Your AI assistant keeps answering from documents that were changed or deleted weeks ago, confidently and with no error, because its search index is a snapshot that quietly fell behind the source. The fix is to keep the index in step with the source as it changes, and to measure how fresh it is instead of trusting last night's rebuild to have caught everything.

DomainData & Retrieval
OutcomeNothing goes stale unnoticed: only changed content is re-processed, deletions take effect in minutes instead of overnight, and how fresh the index is becomes a number on a dashboard rather than a hope.
Stack Change data captureIncremental re-embeddingDelete propagationFreshness SLAIndex-lag monitoring

An engineer asks the internal assistant how to schedule a retry, and it answers with total confidence: call queue.enqueueWithBackoff(). The code looks right, passes a glance, gets merged. Then it fails in review, because that function was deprecated two releases ago and deleted last month. The assistant had answered from documentation the company already replaced, and nothing in the system knew the difference.

Nobody was wrong, exactly. The retriever did its one job, fetch the closest passage, and the closest passage was the old reference page, which still sits in the vector index weeks after the source deleted it. The model summarized what it was handed, like a good assistant. The bug was not in the model. It was in the missing wire between the source changed and the index caught up.

Why it goes stale

A RAG index (retrieval-augmented generation, where the assistant answers by first searching your documents and pasting the matches into the model’s prompt) is not a library. It is a cache of your data, a snapshot taken at ingest, and like every cache it drifts from the source the moment the source moves. The demo indexes once and looks perfect. Production data changes hourly, and the index does not follow unless you make it.

Vector search makes the drift invisible. It returns the passage whose meaning is closest to the question, and closeness has no idea the page it is handing back was superseded yesterday or deleted last month. A stale chunk and a current one look identical to a similarity score. So the engine confidently retrieves a removed function’s docs because they are, semantically, a perfect match for a question about retries.

And staleness fails silently, which is what makes it dangerous. The usual quality check, a faithfulness eval, scores the answer against the passage that was retrieved: was the answer supported by its context? A confidently wrong answer built on a stale passage passes that test with full marks, because the answer really is faithful to the outdated text. The model did not hallucinate. The retrieval layer served old evidence, and nothing in the stack raised a hand.

A stale index serves a confident wrong answer that the eval still passes The source of truth updated its docs and deleted the old page. The vector index still holds the old page and has not embedded the new one. A query retrieves the stale page, the model answers by recommending the deleted function, and a faithfulness eval passes because the answer matches the stale passage it was given. The index is a cache nobody invalidated. Source of truthnew docs shipped,old page deleted Vector index (stale) removed function page, still here current page, not yet embedded index lag: 34 days and counting Modelanswers from it "call enqueueWithBackoff()"deleted two releases ago faithfulness eval: PASSit matches the stale passage IN RAG: the eval scores the answer against the passage it was handed, not against the truth, so staleness sails through green.
Similarity does not know what is current. The retriever ranked by meaning and returned a page the source had already deleted, and the one test most teams run waved it through.

The failure mode nobody handles is deletion. Adding and updating documents is the easy path, the one every ingestion tutorial covers. But when a source doc is deleted, a naive pipeline leaves its chunks floating in the vector store as orphans, and an orphan is worse than a gap. As one engineer put it on r/Rag, a stale chunk that never gets removed is worse than one that never got indexed, because now retrieval has two conflicting answers and picks by similarity. For a codebase, where every commit changes something, this is not an edge case, it is the default failure.

Cron, poll, or capture: only one holds

When a stale index bites, teams reach for one of three sync strategies. Two are half-measures.

Re-index everything, nightly. The default. A cron job rebuilds the whole index after dark. It is simple, and it re-embeds the entire corpus every run, paying to re-encode millions of unchanged tokens to catch the handful that moved. Embeddings are cheap per token, so this hides for a while, then scales into real money and real hours. Worse, it leaves a staleness window of up to a full day, and a deletion at the source does not take effect until the next crawl notices the document is gone.

Poll and diff. Better: re-read the sources on a schedule, compare against what you indexed, update only what changed. But your staleness is now bounded by your poll interval, not by how often the source actually changes, and diffing large sources means heavy repeated scans just to learn that almost nothing moved.

Capture the change at the source. The one that holds. Tap the source’s own change stream so an insert, an update, or a delete becomes an event the instant it happens, and the pipeline touches only the delta. For a database, that stream already exists: change data capture (CDC) reads the write-ahead log the database keeps for its own recovery and emits every row mutation, in order, deletes included.

The honest caveat is that event-driven is not automatically right. A document that changes twice a year does not need a streaming pipeline; a nightly batch is the correct, boring answer for a static archive. And CDC is not free to run: tapping the write-ahead log means standing up and monitoring a streaming stack, and the log carries every column, so you filter the sensitive fields out before they ever reach an embedding. The rule is to match the mechanism to how fast the data actually moves, and for the operational data that RAG assistants answer from, it moves fast.

Three sync strategies and the staleness each leaves Three rows. Nightly full re-index re-embeds everything and leaves up to a day of staleness with late deletes. Poll and diff bounds staleness by the poll interval with heavy scans. Change data capture turns every insert, update, and delete into an event and touches only the delta, the recommended choice for operational data. Your cron sets your staleness. Your source should. Nightly full re-indexre-embeds the whole corpus every run · up to a day stale · deletes land at the next crawlwasteful Poll and diffstaleness bounded by the poll interval, not the source · heavy repeated scans to find littlehalf-measure Capture the change (CDC)every insert, update, delete is an event the moment it happens · touch only the deltaholds IN RAG: CDC reads the log the database already keeps for crash recovery, so the change feed is data you are not yet using.
Scheduling a rebuild is a workaround for not knowing what changed. Change data capture removes the guesswork: the source tells you, exactly and immediately, what moved.

The build

Here is the reference architecture, the wire between source and index that should have been there.

  1. Detect change at the source. Tap the change stream: CDC (a tool like Debezium reading the Postgres write-ahead log or MySQL binlog) for databases, webhooks for SaaS sources that emit them, content-hash polling for the legacy systems that emit nothing. The output is one ordered stream of inserts, updates, and deletes.
  2. Queue the change. Land those events on a durable log (Kafka or equivalent) so they replay after a failure, retry safely, and stay in order. Order matters: a delete must not overtake the update it was meant to follow, or you resurrect a deleted doc.
  3. Re-embed only the delta. Chunk the changed document, hash each chunk, and compare against the hash you stored last time. Embed only the chunks whose hash actually changed. This, not a cheaper model, is where the embedding bill collapses.
  4. Upsert with version metadata. Write the changed vectors carrying more than their numbers: the source doc id, the chunk hash, a version, source_updated_at, and indexed_at. That metadata is as load-bearing as the embedding, because it is how you prove what was current when a passage was served.
  5. Propagate deletes as first-class events. A source delete becomes a delete-by-id; a whole document going away becomes a delete-by-filter over its chunks. Re-chunking is a delete of the old chunk ids followed by an insert of the new. Never an orphan.
  6. Measure index lag. Every completed write stamps indexed_at minus source_updated_at. That single number is your freshness, and the next section makes it an SLA.
The RAG freshness pipeline, source change to fresh index A change at the source is captured by CDC and lands on a durable change log. An incremental worker hash-diffs the chunks, embeds only the changed ones, upserts them with version metadata, and deletes obsolete chunk ids from the vector store. A freshness monitor records index lag as indexed-at minus source-updated-at. Detect the change, touch only the delta. SourceDB / SaaS / files CDCWAL / binlog Change logordered, durable Incremental worker hash-diff chunks · skip unchanged embed only the changed chunks upsert + delete obsolete ids Vector storeversion metadata Freshness monitorindex lag = indexed − updated IN RAG: a delete rides the change log like any edit and lands as a delete-by-id in the store, never a chunk left floating after its source is gone.
One change in, one delta out. The source tells the pipeline exactly what moved, the worker re-embeds only that, and deletes travel the same path as edits, so the index tracks the truth instead of a nightly snapshot of it.

What the frameworks give you, and where they stop

You do not have to build the middle from scratch. Two frameworks ship the dedup-and-cleanup core, and reading what they actually do, rather than what the docs imply, is the difference between a build that holds and one that quietly leaks orphans.

LangChain’s indexing API keeps a record manager, a small SQL ledger mapping each chunk to a content hash. On every run it hashes the incoming chunks, skips the ones whose hash it has already seen, and refreshes their timestamp so cleanup spares them. Its cleanup modes are where deletion lives: incremental removes the stale vectors for the source ids it saw this run, full sweeps everything not refreshed. The catch, stated plainly in its own source, is that there is no tombstone object and the record manager is a separate store from the vector database. So a partial failure splits the two: the ledger write lands, the vector write fails, and now the record of what is indexed disagrees with what actually is. And cleanup=None deletes nothing, which means the convenient default leaves orphans.

LlamaIndex’s ingestion pipeline solves the same problem with a docstore keyed on a document hash: a changed doc triggers a delete of its old nodes and a re-embed of the new ones, and its UPSERTS_AND_DELETE strategy also sweeps documents that vanished from the source. Its own trap is quieter: attach no vector store and it silently downgrades to duplicates-only, doing none of the deletion you thought you configured, and a few store integrations cannot delete by document id at all, so their orphans never leave.

Underneath both, the vector store is blunt about what it owns. pgvector has no document-lifecycle layer: it gives Postgres a vector type, and upsert, delete, and versioning are ordinary SQL you write, with the wrinkle that vacuuming an HNSW index after a big delete is slow enough that reindexing first is the recommended path. Qdrant exposes delete-by-id and delete-by-filter directly, with write consistency you tune from eventual toward stronger. None of them makes freshness automatic. They give you the primitives; the pipeline is yours.

What each sync framework gives you, and where it stops A four-row comparison. LangChain indexing gives hash-skip and cleanup modes but has no store tombstone, can desync its ledger from the store, and leaves orphans when cleanup is off. LlamaIndex ingestion gives docstore-hash upsert and a delete sweep but silently downgrades to duplicates-only with no vector store and some integrations cannot delete by document. pgvector gives raw SQL upsert and delete but has no document-lifecycle layer and slow HNSW vacuum after deletes. Qdrant gives delete-by-id and delete-by-filter but is eventually consistent by default and leaves versioning to your metadata. Primitives, not freshness. the dedup-and-cleanup core is off the shelf; the trap is what each leaves behind FRAMEWORK GIVES YOU WHERE IT STOPS (the orphan trap) LangChainindexing API hash-skip + cleanup modes no store tombstone; ledger and store can desync;cleanup=None leaves orphans LlamaIndexingestion docstore-hash upsert + delete sweep downgrades to dedup-only with no vector store;some integrations can't delete by document pgvector raw SQL upsert & delete no document-lifecycle layer (you write it);HNSW vacuum after big deletes is slow Qdrant delete-by-id / by-filter eventual consistency by default;versioning is metadata you design IN RAG: the dedup-and-cleanup core is off the shelf; delete propagation and the freshness SLA are still yours to build.
They hand you primitives, not freshness. Every one ships the easy half, content-hash dedup, and stops at the hard half, reliable delete propagation, which stays your job.

Freshness is an SLA, not a hope

The point of all this is a number you can put on a dashboard and defend: index lag, the time between a change at the source and that change being live in the index, measured as indexed_at minus source_updated_at. Report its 95th percentile, not its average, because the average hides the one document class that lags for hours.

Then set the target by what staleness costs. A pricing or policy change is wrong in minutes and has to be fresh in minutes; product docs can trail by hours; an archive nobody edits can trail by a day without harm. Those tiers are engineering examples, not a standard, but the discipline they encode is the whole point: freshness is a service-level objective per data class, monitored and alerted like Kafka consumer lag, not a side effect you hope the nightly job delivered. An orchestrator that understands data freshness, rather than only “did the job run,” is what turns the target into an alarm when it slips.

Index lag as a freshness SLO, with example targets by data class Index lag is defined as indexed-at minus source-updated-at. Example service-level targets by data class: pricing and policy in minutes, product docs in hours, archives in days. The metric to alert on is the 95th percentile, not the average. Freshness is a number you own. index lag = indexed_at − source_updated_at · alert on the P95, not the mean Pricing / policywrong the instant it changesminutes Product / API docstolerates a short trailhours Archivesrarely edited, low stakesdays IN RAG: pick the SLO by what a stale answer costs, then alert on the P95 lag before a user hits it.
Set the target by what staleness costs. The tiers are examples, but the move is not: pick a freshness SLO per data class, measure the P95 lag against it, and alert when it slips.

Does it hold?

The difference between a build and a blog post is that a build can be tested, and this one has three unglamorous tests a serious buyer will run.

The staleness test. Change a source document, start a stopwatch, and keep asking the question whose only correct answer is the new version. The answer has to flip within your stated SLO. If it does not, your freshness story is fiction, and the number you quoted is decoration.

The deletion test. Delete a source document and confirm its chunks are gone from retrieval, not hidden behind a filter, gone. Then change a document’s chunking and confirm the old chunks left with it. Orphans are invisible until the day the deleted answer comes back, so you hunt them on purpose.

The lag dashboard. P95 index lag per data class, on a screen, with an alert wired to the SLO. If you cannot show the number, you do not have a freshness SLA, you have a nightly job and a hope, and the two look identical right up until the assistant recommends a function you deleted.

That is the whole discipline: the index is a cache, so treat it like one. Detect the change at the source, re-embed only the delta, propagate deletes as first-class events, and measure the lag. It is the freshness half of the problem a RAG evaluation harness cannot see, because a faithfulness eval passes a stale answer. And it is the retrieval-layer cousin of the agent that returns 200 OK and is still wrong, leaning on the same scattered-source reality as no model outruns a silo. A confident answer from an index nobody kept current is not a smarter model away from being right. It is a pipeline away.