Gracient.ai
Start a conversation
← Articles
July 12, 2026 · 8 min read #AI#RAG#Retrieval#Search

The Answer's in Your Docs. Your RAG Can't Find It.

Your RAG assistant gives a vague non-answer, and the maddening part is that the real answer was sitting in your documents the whole time. It didn't miss because the model is dumb. It missed because search is a funnel with three leaks: the answer got chopped up when you filed it, the search matched the wrong words, and the right passage got ranked too low to make the cut.

Someone asks your documentation assistant a simple question: “What’s the rate limit on the /v2/upload endpoint?” The number is right there in your API reference, in a table, under a heading that literally says upload limits. The assistant comes back with a confident paragraph about uploads that never states the number.

You check. The document is indexed. The answer exists. And the model, when you paste the right passage in by hand, answers perfectly. So the model isn’t the problem, and the data isn’t missing. Retrieval simply failed to hand the model the one passage it needed.

This is the most common and most frustrating failure in production RAG, and almost everyone misdiagnoses it. They swap the embedding model, or upgrade to a bigger LLM, and the miss stays exactly where it was. Because the miss isn’t in the model. It’s in the pipe that feeds it.

Here’s the reframe that fixes how you debug this. Retrieval is not one operation. It’s a two-stage funnel. The first stage, retrieve, casts a wide net and pulls maybe a hundred candidate passages out of your whole corpus; its only job is recall, getting the right passage into that candidate set at all. The second stage, rank, sorts those candidates and forwards the top handful to the model; its job is precision, getting the right passage to the top.

Two stages, two different jobs, and three places the water leaks out. The answer can get chopped up when you chunked and indexed it. The search can match the wrong words and never pull it into the candidate set. Or the right passage can get pulled in but ranked too low to survive the cut. The crucial thing is that two of those three leaks happen before ranking, which is exactly why buying a fancier reranker so often changes nothing.

Retrieval is a two-stage funnel that leaks in three places A left-to-right funnel. A query enters a Retrieve stage whose job is recall, producing a candidate set of about 100 passages, then a Rank stage whose job is precision, producing the top 5 passages sent to the LLM. Three leaks are marked: chunking and matching sit at the Retrieve stage, ranking sits at the Rank stage. A funnel with three leaks. question/v2/upload? RETRIEVEjob: recallcast a wide net ~100 candidatesthe shortlist RANKjob: precisionsort, keep the top 5 LLMtop 5 leak 1: chunkingleak 2: matchingleak 3: ranking IN RAG: two of the three leaks happen before ranking, which is why a fancier reranker so often fixes nothing.
Retrieve casts the net; rank sorts the catch. Debugging RAG means finding which of the three leaks you actually have, because the fix for one does nothing for the others.

Leak 1: the answer got chopped up

Before anything can be retrieved, your documents get split into chunks and embedded. That splitting is where the first answers die. Chunk on a blind character count and you slice a table clean away from the heading that gave it meaning, so the chunk holding your rate-limit number no longer says a word about /v2/upload. Or you go the other way, a single 2,000-token mega-chunk covering install, config, networking, and limits, and the embedding becomes a mush-average of six topics that is a strong match for nothing.

The first fixes are cheap. Chunk on structure, headings, sections, code blocks, not a fixed window, so a table stays with its title. Add overlap so an answer straddling a boundary survives in at least one chunk. And when a chunk is still too orphaned to make sense alone, prepend a sentence of context to it before you embed it, so “the rate limit is 100 requests per minute” carries “this is the /v2/upload endpoint reference” with it. Anthropic’s version of this, contextual retrieval, cut failed retrievals by 35% on its own benchmark, by 49% once paired with keyword search, and by 67% once a reranker was added on top, for a one-time indexing cost of about a dollar per million tokens. Cheap insurance against the leak that starts before search even runs.

A blind chunk cut splits a table from its heading; structure-aware chunking keeps them together On the left, chunking on a blind character count cuts between the heading slash v2 slash upload rate limits and the value 100 requests per minute, so the chunk with the number no longer mentions the endpoint and cannot be matched. On the right, chunking on structure with overlap keeps the heading and the value in one chunk, so the number keeps its context. Chunk blind, and the answer loses its heading. BLIND CHARACTER CUT ## /v2/upload rate limits chunk boundary 100 requests / minute the number-chunk never names /v2/upload → miss CHUNK ON STRUCTURE + OVERLAP ## /v2/upload rate limits100 requests / minute the number keeps its heading → hit IN RAG: a table split from its title becomes a chunk that no longer looks like the answer to anything.
Where the answer dies first: the split. Cut on a blind character count and the number loses the heading that made it findable; chunk on structure, with overlap, and it keeps the words a search would look for.

Leak 2: the search matched the wrong words

Say the chunk is perfect and whole. It can still miss, because of how the default search works. Dense vector search embeds your query and every chunk into a space where nearness means similar in meaning, and meaning is exactly the wrong lens for an exact identifier. To an embedding model, /v2/upload looks like generic “upload,” ERR_CONN_RESET_481 looks like “a network problem,” and a part number looks like noise. It confidently retrieves passages that are about the right topic and skips the one with the literal string you need.

The old-fashioned answer is the fix: BM25, plain keyword search, which does nothing but match terms and therefore nails /v2/upload on the exact token an embedding fumbles. You don’t choose between them, you run both and fuse the results, a setup called hybrid search. The standard fuser is reciprocal rank fusion, which just adds up 1 / (rank + k) for each result across the two lists so a passage both methods like floats to the top. One caveat worth knowing: some libraries advertise “fusion” that is really just concatenate-and-dedupe under the hood, so check that yours actually does the rank math.

Dense search misses an exact identifier that keyword search nails For the query slash v2 slash upload rate limit: dense vector search returns generic upload pages and misses the exact endpoint, marked as a miss. BM25 keyword search matches the exact string and returns the right page, marked as a hit. Hybrid search fuses both and returns the right page. Similarity is not the same as the exact words. query: /v2/upload rate limit Dense (vectors) reads it as generic "upload" returns: uploading files 101, upload best practices misses the number BM25 (keywords) matches the exact string returns: /v2/upload reference, limits table finds the number Hybrid (fuse both) meaning AND exact terms RRF floats the shared winner to the top gets it, robustly IN RAG: dense reads /v2/upload as generic "upload"; keyword search nails the exact string; hybrid keeps both.
Meaning search and word search fail on opposite things. Dense vectors miss the literal token; keyword search misses the paraphrase. Running both and fusing them is why hybrid is now the production default.

Leak 3: the right passage got buried

Now the good chunk is whole and it’s in your candidate set. You can still miss, because being retrieved isn’t the same as being ranked first. First-stage search sorts by how close two vectors are, which is a fast, rough stand-in for real relevance, not the real thing. Your rate-limit passage comes back at rank 14. You forward the top 5 to the model, and it never sees it.

The fix is to retrieve many, then rerank to few. Cast a wide net first: pull the top 50 or 100 candidates instead of 5, so the right passage is almost certainly somewhere in the pile even if it’s sitting near the bottom. Then re-sort that pile with a slower, sharper judge, a cross-encoder reranker.

Here’s why that second pass finds what the first one couldn’t. Fast search never actually reads your question and a passage together. Ahead of time, it boils the question down to a single point and every passage down to its own point, then just measures how close the two points are, so it’s really grading each passage in the abstract, without your specific question sitting next to it. That’s how a spot-on passage ends up at rank 14. A cross-encoder does the opposite: it takes your question and one candidate and reads them side by side, in the same pass, then scores how well that passage answers that question. Reading the pair together is slower, because it re-runs the model on every candidate, but it’s exactly the head-to-head comparison that lifts a buried passage to the top.

Two cautions from the real implementations. The reranker runs a full model pass over every candidate, so it adds real latency and cost. And most rerankers quietly cut their input off at 512 tokens, so an oversized chunk gets truncated before it’s even scored, which is one more reason the bloated chunks from Leak 1 come back to bite you.

Bi-encoder compares separate embeddings; cross-encoder reads the pair together On the left, a bi-encoder embeds the query and the document separately into two vectors and scores them by cosine distance, so the two never interact. On the right, a cross-encoder feeds the query and document into one model together and outputs a single relevance score, reading them jointly. The bi-encoder is fast for first-stage retrieval; the cross-encoder is slower but more accurate for reranking. Why the reranker catches what search missed. BI-ENCODER · retrieval query → vec doc → vec compare by cosine distance the two never meet; fast, approximate CROSS-ENCODER · rerank (query, doc) → one model → score reads the pair together slower, but sees the real relevance IN RAG: a bi-encoder measures a distance between two points; a cross-encoder actually reads the query and the passage in the same breath.
The difference is whether the query and the passage ever meet. Retrieval keeps them apart for speed; reranking puts them in the same forward pass, which is exactly the interaction that surfaces a buried answer.

Which leak is actually yours?

Three leaks, three different fixes, and here’s the mistake that wastes the most time: bolting a reranker onto a recall problem. A reranker can only reorder passages it was handed. If the right one never made the candidate set, no amount of reranking will conjure it.

So run one test before you change anything. Take a handful of questions you know the answer to, note the exact passage that answers each (this is the seed of a real retrieval eval set), and retrieve a big candidate set, the top 100. Then ask one question: is the gold passage in there at all? If it’s missing, you have a recall problem, and the fix is upstream, chunking or matching. If it’s present but sitting at rank 40, you have a ranking problem, and that’s when a reranker earns its keep. One check, and you know which hole to plug instead of guessing.

One test tells you whether you have a recall problem or a ranking problem Retrieve the top 100 candidates and ask whether the known correct passage is in the set. If it is absent, it is a recall problem, fixed upstream by chunking and matching, and a reranker cannot help. If it is present but ranked low, it is a ranking problem, fixed by a reranker. One question tells you which leak. Is the gold passage in your top 100?retrieve wide, then look NO → recall problemfix upstream: chunking / matching. a reranker can't help. YES, but ranked low → rankingadd a cross-encoder reranker. now it earns its keep. IN RAG: a reranker can only reorder what retrieval already found, so never point it at a recall problem.
Diagnose before you fix. The single most useful number in retrieval engineering is whether the right passage was retrieved at all; it splits every miss into recall or ranking and points you at the correct half of the funnel.

The pipe, not the model

Notice what never came up: a bigger model, a newer embedding, a longer context window. Our rate-limit answer was in the corpus the whole time. It missed because a table got split from its heading, because meaning-search skimmed past a literal path, and because a good passage got out-ranked by plausible neighbors. Three holes in a pipe, each with a boring, known patch: chunk on structure with overlap, search hybrid and fuse the results, and retrieve wide then rerank narrow, after you’ve checked which leak you actually have.

That is the unglamorous truth about RAG that works. The intelligence you’re paying for on the generation side is wasted if the retrieval side hands it the wrong page, and almost none of the wins are on the generation side. This is the half of RAG we build at Gracient: the retrieval pipe that reliably puts the right passage in front of the model, so the answer that was always in your documents is the one your users actually get. Let’s talk.

Read next

AI Jul 1, 2026

No Model Can Outrun a Silo

Enterprise AI keeps stalling, not on the model, but on data trapped in silos. Why that happens, thirty years of fixes that didn't stick, and what actually makes data AI-ready.

Read →
AI Jul 11, 2026

Your Agent Remembers Too Much

An AI agent working a long task stuffs everything it sees into its short-term memory, until the memory is full of junk and it forgets what it was doing. The fix is not a bigger memory. It is teaching the agent what to throw away: keep the goal and the facts that matter, summarize the rest, and file the details somewhere it can look them up later.

Read →