Gracient.ai
Start a conversation
← Case studies
AI retrieval & evaluation · Reference build · July 2, 2026

A RAG evaluation harness that catches what the demo hides

A reference build for measuring RAG quality instead of eyeballing it: a fixed test set, the finding and the writing scored separately, an automatic gate that blocks regressions, and live monitoring that feeds failures back in. The point is to tell a retrieval problem from a generation one, and ship on evidence.

DomainAI retrieval & evaluation
OutcomeRetrieval regressions caught before they ship; every failure traced to retrieval or generation
Stack Golden eval set (silver to gold)RAGAS / DeepEvalRetrieval metrics: recall@k, MRR, nDCGFaithfulness + answer relevanceHybrid search + rerankingCI eval gate + production tracing

A RAG demo is easy to build. You split your documents into small pieces, store each piece as a searchable numeric fingerprint of its meaning (a vector), and let the model answer using the closest matches. (RAG just means the model looks things up in your documents before it answers.) It sails through your ten test questions. Then real users arrive with real questions, and it starts to crack.

The problem is that a basic setup often pulls up the wrong documents, by some accounts around 40% of the time (the exact number varies by setup). And the model will not stop to warn you. It is built to be helpful, so it writes a clean, confident answer from the wrong source. It looks right and is quietly wrong. Most teams never notice, because they only read the final answer, not what went into it. So the goal of this reference build was simple: make RAG quality something you can measure, not something you eyeball.

The idea

One rule drives everything: measure the finding and the writing separately.

A RAG system does two jobs. First it finds the right documents (retrieval). Then it writes an answer from them (generation), staying faithful to what it found.

A single “was the answer good?” score cannot tell you which job failed, and the two need opposite fixes. Finding comes first and sets the ceiling: if the right document never gets pulled up, no prompt tweaking will save the answer.

So the harness scores the two jobs on their own, and blocks a release if either one slips.

The evaluation harness A golden set of question, answer and source cases feeds the RAG pipeline (split, retrieve, generate). The output is scored on two separate cards: a retrieval score (recall, precision, ranking) and a generation score (faithfulness, answer relevance). Both feed an automatic gate that blocks the merge if a metric drops. A dashed loop returns failures to the golden set, and the same scorers also run on sampled production traffic. Measure retrieval and generation separately. Golden setversioned Q + answer + sourcesilver → gold (expert-reviewed) RAG pipelinesplit · retrieve · generate Retrieval scorerecall · precision · ranking Generation scorefaithfulness · answer relevance Auto gateblock the merge ifa metric drops low-scoring cases feed back into the golden set The same scorers run on sampled production traffic, with tracing to show which step dropped the right piece.
Two scorecards, one gate. The finding and the writing are graded on their own, and every change reruns the golden set before it can ship.

The golden set, and a judge you can trust

The harness is only as good as the test set behind it. We build a golden set: a fixed, versioned list of real questions, each with the answer it should give and the source that answer should come from. We draft it quickly with an AI language model (LLM) as the “silver” version, then have someone who knows the domain check and fix it into the trusted “gold” version.

Quality beats volume. A few hundred real, hard questions beat thousands of near-identical made-up ones.

The grading is done by an LLM, so we treat that grader like a lab instrument: the same model every time, set up to give the same score for the same answer, and checked against human scores before we trust it.

A golden set and a judge you can trust Your documents are drafted by an LLM into a silver set of question-answer pairs, reviewed by a subject-matter expert into a gold set of verified, versioned cases, which the LLM judge grades against. The judge is pinned to a fixed model at temperature zero and validated against human labels before it is trusted. A golden set, and a judge you can trust. Your documents Silver setLLM-drafted Q + A Gold setexpert-verified · versioned LLM judgepinned model · temp 0 draft expert review grades validated vs human labels first
Quality over volume. A few hundred hand-verified cases beat thousands of synthetic near-duplicates, and the LLM that grades them is pinned and checked against human labels before you trust a single score.

The two scorecards

The retrieval scorecard asks: did we pull up the right documents? Recall asks whether we got everything the answer needs. Precision asks how much of what we pulled was useful, not noise. Ranking asks whether the best evidence landed near the top, not buried on page two.

The generation scorecard asks: is the answer any good? Faithfulness checks that every claim is backed by the documents we retrieved. Relevance checks that it actually answers the question, correctly.

How high you set the bar is your call: a help-desk FAQ can live with the odd slip; a medical or legal system cannot.

Reading them together

Here is where keeping the two scores apart pays off, because the same wrong answer can come from two opposite problems.

Is it retrieval or generation? A two-by-two grid. Columns are faithfulness (low on the left, high on the right); rows are answer correctness (correct on top, wrong on the bottom). Top-left, correct but not grounded, is a grounding failure. Top-right, correct and grounded, is good. Bottom-left, wrong and ungrounded, is both. Bottom-right, wrong but faithful to context that lacked the facts, is a retrieval failure. Is it retrieval or generation? the same wrong answer, two opposite causes low faithfulnesshigh faithfulness answercorrect answerwrong Grounding failureright, but not from the contextfix: prompt, citations, guardrails Good ✓grounded and correct Bothwrong and ungrounded Retrieval failurefaithful, but context lacked the factsfix: retrieval
Same symptom, opposite fix. Bottom-right: the model used the retrieved context faithfully, but that context was the wrong passage, so the answer is wrong. That is retrieval's fault, not the model's. Top-left: the answer drifted from good context, and that is the model's. One end-to-end score cannot tell these apart; two can.

The scores tell you the fix

You measure so you know what to change. Each low score points at a specific fix, not a guess.

The retrieval score tells you the fix Two mappings. Low context recall maps to retrieve more, hybrid search (keyword plus vector), and re-splitting. Low context precision maps to adding a reranker to re-score the top hits. The retrieval score tells you the fix. Low context recallretrieve more · hybrid search (keyword + vector) · re-split Low context precisionadd a reranker to re-score the top hits
Diagnosis, then treatment. Low recall is a coverage problem, fixed by fetching more and fetching smarter; low precision is a noise problem, fixed by a reranker, a second pass that re-sorts the top hits by relevance. Reported gains are real but modest (hybrid adds roughly 5 to 15 points of recall, a reranker roughly 10 to 15 of precision), so treat vendor claims of "50 to 100%" with suspicion.

The toolkit that usually results is familiar: keyword search combined with vector search, a second-pass reranker that re-sorts the top few results by how relevant they really are, and smarter splitting so each piece carries enough context to be found later.

Which of these you actually need is decided by the scores, not by fashion.

Outcome

The guarantees are built in, not promised. Every change is scored against the golden set before it ships, so a regression is caught automatically, not by a customer, and a wrong answer tells you within seconds whether the problem was finding or writing. The same checks run on live traffic too, so quality drift lands on a dashboard, not in a complaint. You ship on evidence.

That harness is work we can build with you.