Systematic Debugging of RAG Retrieval Quality

I. First, recognize one fact: when RAG answers poorly, retrieval is at fault 90% of the time

When RAG answers poorly, many people's first reaction is "use a larger model," "change the prompt," or "add rerank." That is blind tinkering without locating the problem.

RAG = retrieval + generation. No matter how strong the generation model is, if the context fed to it is garbage, the output will be garbage too -- Garbage in, garbage out. Whether you use GPT-4o or Claude, if you give it a chunk that has nothing to do with the question, it will either make things up or honestly reply, "I don't know."

So the first ironclad rule of debugging RAG is: prove retrieval works before changing generation.

How do you prove it? Surface every intermediate result. Do not stare only at the final answer and guess what failed.

II. Systematic Triage: A Five-Layer Fault-Isolation Method

RAG is a pipeline, and every layer can fail. Investigate it layer by layer; do not skip ahead.

User query
   │
   ▼
① Recall (which chunks were retrieved)────────── recall layer
   │
   ▼
② Similarity scores (are the top-ranked ones really more relevant?)── ranking layer
   │
   ▼
③ Order after reranking (does rerank help or hurt?)── reranking layer
   │
   ▼
④ Context fed to the model (was it truncated or reordered?)── assembly layer
   │
   ▼
⑤ Model output (did the model actually use the context?)────── generation layer

Diagnose from top to bottom, asking at every layer: "Is this layer's output correct?" The problem is in whichever layer is wrong. Do not discover an incorrect answer at layer five and then blindly tweak layer two -- you do not know which layer failed. The following sections explain how to distinguish them.

① Inspect recall: what exactly did retrieval return?

Print the original text of the top-k chunks and inspect it manually: is a chunk relevant to the question present?

• Yes -> recall is fine; investigate further downstream.
• No -> recall has failed; check chunking, embedding, and filters (see Section III).

This is the lowest-cost, highest-return debugging action. Unfortunately, many people skip it and go straight to agonizing over the final answer.

② Inspect similarity scores: are the top-ranked results truly more relevant?

Was a relevant chunk retrieved but ranked outside top-k and cut off? That is a ranking problem.

Print the score for every retrieved chunk and inspect the distribution. Under normal conditions, relevant chunks should score clearly higher than irrelevant ones. If relevant and irrelevant scores are bunched together (for example, all between 0.70 and 0.72), the embedding lacks discriminative power -- either the model is inadequate, or the query and doc are not in the same vector space (often because an instruction prefix was forgotten for the query).

③ Inspect the order after reranking: is rerank helping or hurting?

If you added rerank, compare the order before and after reranking. Rerank should move more relevant chunks forward. If it pushes a relevant chunk that was originally ranked first down to fifth and promotes irrelevant content instead, rerank is hurting rather than helping; usually the rerank model is a poor fit for your domain.

If you have not added rerank, do not rush to add it. First confirm that the preceding layers are sound.

④ Inspect the context fed to the model: did assembly go wrong?

Reaching this step means the preceding layers are correct but the answer is still wrong. Inspect the context actually assembled into the prompt:

• Was a relevant chunk cut off by top-k?
• Was half of it removed because of a token limit?
• Is the chunk order reasonable? (In some cases, the most relevant content should come first.)
• Are chunks duplicated? Has a complete semantic unit been split in two?

⑤ Inspect the model output: did the model actually use the context?

Only at the end does the generation layer come into play. If everything before it is correct and the model still answers incorrectly, check whether the prompt strictly requires the model to "answer only from the context." A loose prompt lets the model fabricate answers from its own parametric knowledge, which is exactly where hallucinations take root.

III. No Recall: The Most Common "Retrieval Failure"

"No recall" has three distinct root causes, each with a different diagnostic method.

Chunking destroys semantics. A complete fact is split across two chunks; the query matches only one half, while the other is lost. Diagnosis: check whether retrieved chunks quote content out of context. Solution: chunk semantically (by paragraph or heading), rather than mechanically by character count; preserve overlap; or use small chunks plus a parent-child chunk (parent document retrieval) strategy.

Embedding mismatch. The query and doc use different embedding models, or they use the same model but the query lacks an instruction prefix (many embedding models use different prefixes for query/doc). Diagnosis: take a known relevant query-doc pair and inspect its similarity. If a person considers it relevant but the model assigns a low score, embedding is the culprit.

Filters are too strict. You added metadata filters before retrieval (by time, category, or permissions) and filtered out all relevant chunks. Diagnosis: temporarily disable every filter and see whether recall returns. If it does, the filters are the cause.

IV. Retrieved but Not Used Correctly (Precision): Fetching Too Much

Relevant chunks are in the retrieval results but are drowned in noise, leaving the model unable to determine which source to trust.

Missing rerank. Pure vector retrieval has limited precision, and half of the top-10 may be irrelevant. This is where rerank is genuinely useful -- provided you have confirmed recall is not the problem.

top-k is too large. Set k too high, and noise floods the context, lowering answer quality instead. Bigger k is not always better; run an ablation from k=3 to k=20 and see which setting is most accurate.

No hybrid retrieval. Pure vectors are insensitive to keywords and proper nouns -- a product model number or a person's name in a query can leave vector retrieval at a loss. Add BM25/keyword retrieval for hybrid search, and recall can improve immediately.

V. Retrieved but Ignored by the Model: The Prompt Is at Fault

Retrieval is correct, the context contains the right answer, and the model still answers incorrectly. The prompt is to blame.

Common problems:

• The prompt does not say, "Answer only from the provided context; if the context does not contain the answer, say you don't know.";
• The context is not placed prominently, or it is separated from the question by a pile of other instructions;
• You ask the model to "synthesize" multiple chunks, and it fills in the gaps on its own.

Diagnosis: give the model the same context + question directly, without retrieval, and see whether it can answer correctly. If it can -> the RAG pipeline's context assembly is the problem; if it cannot -> the prompt itself needs revision. This is the key experiment for isolating retrieval from generation.

VI. Debugging Toolbox

Several useful tools, ordered by necessity:

1. Retrieval result visualization: print top-k chunks together with their scores. This is the most basic and effective tool. Trace tools such as LangSmith, Langfuse, and Phoenix can all do this.
2. Query-doc similarity comparison: for the same query, manually label which docs are relevant and see whether the embedding score order is correct -- the standard way to evaluate embedding quality.
3. Single-chunk test: take a known chunk containing the answer and put it directly into the prompt for the model. If it answers correctly -> generation is fine and retrieval is the problem; if it does not -> there is a problem at the model/prompt layer. This is the key experiment for isolating retrieval from generation.
4. Bad-case regression suite: save failed cases and rerun them after every change to see whether they are fixed and whether new problems were introduced.

VII. Do Not Optimize Blindly: 90% of "Adding Rerank" Is a Lazy Fix

Here is an unpopular opinion.

I have seen too many teams whose first reaction to poor RAG answers is "add rerank," "use more expensive embeddings," or "switch to GPT-4." That is throwing money at the problem, not solving it.

Rerank is not a silver bullet. If chunking destroys semantics, rerank cannot save it; if filters are too strict, rerank cannot save it; if the prompt has no constraints, rerank cannot save that either. Rerank solves only one kind of problem: recall is fine but precision is insufficient.

That is why the five-layer fault-isolation method matters: locate the problem first, then optimize. Once the location is clear, a very small change -- adjusting chunk granularity, adding a query prefix, or changing one prompt sentence -- can often solve it. You will also know why it improved.

If you blindly pile on components, even if results improve, you will not know which component helped. When it fails again next time, you will still be guessing.

VIII. Make Debugging Reproducible: Add Bad Cases + Traces to a Regression Suite

The biggest debugging trap is treating it as a one-off. You encounter a bad case today, make a change, and it seems fixed; then production breaks again tomorrow. Why? Because you tested only one case.

The correct approach:

1. Every time you find a bad case, save it together with its complete trace (retrieved chunks, scores, rerank order, assembled prompt, and model output);
2. Label the expected answer;
3. Every time you change the pipeline, run the entire regression suite and check whether the pass rate improved or regressed;
4. The larger the regression suite, the more confidence you can have in every change.

This is not complicated, but it turns RAG debugging from "black art" into "engineering." RAG system reliability is built one regression case at a time, not from a single flash of inspiration.

Summary

• When RAG answers poorly, retrieval is at fault 90% of the time. Prove retrieval works before changing generation; do not reverse the order.
• The five-layer fault-isolation method: recall -> similarity -> rerank -> assembly -> generation. Confirm each layer in sequence and fix the layer that is wrong.
• No recall, retrieved but not used correctly, and retrieved but ignored by the model are three different classes of problems, with completely different diagnostics and solutions. Do not conflate them.
• Do not add rerank blindly. Locate the problem before optimizing; a small change often beats a pile of expensive components.
• Add bad cases + traces to a regression suite. Turn debugging from black art into engineering; reliability is built on regression suites.

In one sentence: RAG debugging is not "parameter tuning"; it is "fault localization." Once you locate the problem accurately, half the problem is solved.