Optimizing RAG by Mining Failure Cases

Anyone who has operated a production RAG system has had the same experience: benchmark scores in papers keep rising, yet their own system is a mess once deployed. When users ask questions, the system either gives answers that have nothing to do with what they asked, fails to find information that is clearly in the documentation, or confidently invents an API that does not exist. These errors are failure cases.

But the bad news is that most teams handle failure cases like this: a user complains → an engineer tweaks the prompt based on intuition → the change goes live → the next problem arrives, and they tweak it again. Nothing is retained, classified, or regression-tested. The result is a system that lurches back and forth: a problem fixed today is broken again next week by another "optimization."

This article explains a feedback loop I have successfully used in production systems: from collecting and classifying failure cases through fixing them and incorporating them into an eval set. It is not window dressing; every layer consists of actionable engineering work.

I. Failure Cases Are the Gold Mine of RAG Optimization

First, an uncomfortable truth: public benchmarks offer almost no useful guidance for your system.

The reason is simple. Queries in leaderboards such as MS MARCO, BEIR, and MTEB are "standard retrieval questions," while their document sets are clean, English-language, and domain-agnostic. Your system, however, must handle long Chinese-language documents, users asking questions in fragments, typos mixed into queries, requests that require citing a particular row in a table, and answers that should be "yes/no" rather than a paragraph summary. An entire real world separates the two.

More importantly, benchmarks measure an "average score," while production failures always live in the "long tail." A system scoring 95 might fail every question phrased in a particular way, destroying the user experience. Averages hide disasters.

The real guidance for optimization comes from the failure cases of actual production users. They include the scenario, the intent, and the characteristics of your data. Every failure case is a free, precisely labeled optimization guide. A paper tells you that "reranking improves the average score by 3 points"; a failure case tells you that "recall fails 60% of the time when users ask type X questions, and that is the problem you need to solve."

In one sentence: benchmarks determine your ceiling, while failure cases determine your floor. Raise the floor first, and only then does the average score matter.

II. Collection: Capture User Pain in Telemetry

The first step in failure-case optimization is not fixing problems but collecting them. Without systematic collection, optimization is guesswork. I recommend instrumenting at least the following signals.

First, explicit feedback. Put 👍/👎 controls or a 1-5 star rating beneath every answer. This is the most direct signal. However, users are far less likely to click 👎 than they are to be dissatisfied; most people simply close the page when they are unhappy. Explicit feedback is therefore a "strong signal with low recall." Use it to capture important cases, but not as your only metric.

Second, low-CSAT sessions. Pull complete sessions rated 1-2 and inspect exactly what the user asked, what the system answered, and what context was available. A single session can often yield multiple failure cases.

Third, retry behavior. This signal is underrated. If a user asks a question and then asks it again three seconds later using similar but different wording, you can be almost certain that the previous answer was unsatisfactory. A high retry rate means retrieval or understanding has failed. Mark "query similarity > 0.8 within 10 seconds in the same session" as a suspected failure case and store it automatically.

Fourth, manual sampling. Randomly sample 50-100 production interactions every day and score them manually. This is the gold standard and the baseline for calibrating the three automated signals above. Without manual sampling, you do not even know how low the recall of 👎 feedback is.

What should you record? The key is not to log only the query and answer. You must store the complete intermediate pipeline: the original query, the rewritten query, the list of retrieved chunks with scores, the ordering after reranking, the final context sent to the LLM, the LLM's raw output, and the postprocessed answer. Without this information, a failure case leaves you guessing which stage failed. With it, diagnosis takes minutes.

My approach is to write one structured log record for every request, store it in object storage, and build a query table. Ninety percent of the efficiency of failure-case mining depends on the completeness of your logs.

III. Classification and Diagnosis: Four Distinct Types of Failure Cases

Collecting failure cases does not mean you can fix them. Which stage caused a particular failure? This is the most important step in the entire loop, and also the one most often skipped. I divide failures into four categories, each requiring a completely different remediation strategy.

① Retrieval failed (recall problem)

Symptom: the answer is clearly present in the documentation, but none of the top-k retrieved chunks contains it. This is the most common category.

Diagnostic method: search the knowledge base manually. If you can find the relevant content but it is absent from the system's retrieval results, common root causes include inappropriate chunk size, where chunks that are too small lose context and chunks that are too large dilute relevance signals; an embedding model poorly suited to your domain; query rewriting that drops keywords; or insufficient lexical weighting in hybrid retrieval.

② Found but used incorrectly (precision / utilization problem)

Symptom: the correct passage is present among the retrieved chunks but falls outside the top-k cutoff, or it makes the cutoff but the LLM does not use it and instead relies on a lower-ranked, irrelevant passage.

Diagnostic method: inspect the retrieval list. Where does the correct chunk rank? If it is in the top 5 but the LLM does not use it, the prompt or an overly long context is burying the evidence. If it ranks 6-20, the problem lies in reranking or ranking scores. These require completely different fixes: tune the prompt and context organization for the former, and add reranking or adjust weights for the latter.

③ Generation hallucination (faithfulness problem)

Symptom: the retrieval results are correct, but the LLM cannot resist inventing content that does not appear in the documents. Typical examples include fabricated API names, parameters, and numbers.

Diagnostic method: compare every sentence in the LLM's answer with the context. Any statement unsupported by the context is a hallucination. Common root causes include insufficient prompt constraints, such as failing to emphasize that the answer must be based only on the provided context; irrelevant content in the context that encourages the model to wander; or weak instruction-following by the model itself.

④ Formatting / style error (prompt / postprocessing problem)

Symptom: the facts are correct, but the answer is too verbose, does not follow the requested format, such as returning paragraphs instead of a table, uses mismatched citation numbers, mixes Chinese and English, or has the wrong tone. This category does not affect correctness but seriously harms usability.

Diagnostic method: check whether the answer follows the output requirements stated in the prompt. The root cause is usually an underspecified prompt or a bug in the postprocessing logic.

Once you distinguish these four categories, you will stop using a sledgehammer to crack a nut. I have seen too many teams respond to every failure case by reflexively adding reranking, switching to a larger LLM, or adding more chunks. Most of their problems actually come from chunking and prompts and have nothing whatsoever to do with reranking.

IV. Targeted Fixes: Do Not Add Reranking Blindly

Once failures are classified, each category has corresponding remediation options. The following are directions, not silver bullets. Every fix must be validated with data.

For recall problems (retrieval failed):

• Tune chunking: make chunks smaller or larger, or switch to semantic chunking based on paragraph and heading boundaries. For long documents, try overlapping chunks first to avoid losing information at boundaries.
• Tune queries: add query rewriting or expansion, including synonyms and subquestion decomposition, especially when users ask questions conversationally.
• Tune retrieval: use hybrid retrieval combining vectors and BM25. Lexical matching is critical for proper nouns, model numbers, and code snippets, which pure vector retrieval can easily miss.
• Broaden retrieval: try increasing top-k from 5 to 10, but do not increase it blindly because this adds latency and introduces noise.

For precision problems (found but used incorrectly):

• Add reranking: this is where reranking actually belongs, when recall is sufficient but ordering is inaccurate. Cross-encoder reranking can often move the correct chunk from eighth place into the top 2.
• Reduce context noise: an excessively large top-k dilutes the signal and makes the LLM more likely to follow irrelevant chunks. Test the optimal k; 3-5 is often better than 10.
• Tune how context is presented in the prompt: put the most relevant content first to mitigate the lost-in-the-middle effect, or explicitly state that "the following content is ordered by relevance."

For faithfulness problems (hallucinations):

• Add strict prompt constraints: "Answer only from the context below; if the context does not contain the information, you must answer 'Not mentioned in the documentation'; do not fabricate APIs, parameters, or numbers."
• Require citations: every sentence must cite its source chunk number, forcing the LLM back to the context.
• Add a faithfulness check: use another LLM, or the same one, to check whether the answer can be supported by the context. If it cannot, regenerate the answer or fall back to "Unable to confirm."
• Clean up the context: determine whether irrelevant chunks encouraged the model to wander. If so, return to the precision step.

For formatting problems:

• Provide few-shot examples in the prompt and specify the expected output structure. This is the most effective approach.
• Add structured output using a JSON schema or function calling, turning "style" into "fields."
• Add postprocessing safeguards: fix citation numbering with regular expressions, normalize punctuation, and trim redundant pleasantries.

Watch for one anti-pattern: do not fix one case at a time. Before changing a prompt, ask yourself whether the change will help or hurt other cases. Is there a more general solution? The next section explains how to make this rigorous.

V. Add Cases to the Eval Set to Prevent Regressions

Fixing a failure case does not finish the job. If you simply deploy the fix, you will probably discover two weeks later that the same type of problem has returned or that your change has broken another case. This happens when no eval set acts as a safeguard.

For every confirmed failure case, create a labeled eval case. Each eval case should contain at least:

• query: the user's original question or an equivalent rewrite
• expected_facts: facts the expected answer must contain, extracted manually from the documentation
• expected_chunks: the chunk ids that retrieval is expected to return
• faithfulness: boundaries defining what the answer must not fabricate
• category: which of the four categories above the case belongs to

Turn these cases into an automated eval suite. Run the complete eval set whenever you change the prompt, chunking, retrieval parameters, or model. This provides two benefits. First, it prevents regressions, so fixing A does not silently break B. Second, it quantifies gains, making it immediately clear how many cases a new reranker actually recovers.

Use a tiered eval set: a core set with dozens of hard cases that must all pass, a standard set with hundreds of cases covering common scenarios, and a long-tail set with thousands of edge cases. Run the core set in CI, the standard set weekly, and the long-tail set monthly. A continuously growing eval set is the most valuable asset in your RAG system, worth more than any prompt template.

One more easily overlooked point: the eval set must also contain negative cases that the system should not answer. For example, if a user asks about something outside the knowledge base, the expected response should be "Out of scope," not a fabricated answer. Without these cases, your system will develop the bad habit of answering anything.

VI. The Loop: Production Failure Cases → Classification → Fixes → Eval Validation → Deployment

Connect the steps above to create a sustainable iteration loop:

Production failure cases (feedback/low scores/retries/manual sampling)
    ↓
Replay logs and locate the failing intermediate stage
    ↓
Classify into four types (recall / precision / faithfulness / format)
    ↓
Apply targeted fixes (chunking/retrieval/reranking/prompt/postprocessing)
    ↓
Full eval-set regression + A/B canary rollout
    ↓
Deploy → continuous monitoring → new failure cases enter the next cycle

The core value of this loop is not "fixing one case," but making the system slightly more robust on the real production distribution with every cycle. After six months, your eval set will contain thousands of labeled cases, your classification instincts will have been repeatedly calibrated by data, and new team members will be able to follow this SOP. That is engineering-grade RAG optimization.

Do not skip the A/B canary rollout. Before deploying any change broadly, expose it to 5%-10% of traffic and compare core metrics, including answer accuracy, 👍 rate, retry rate, and hallucination rate. An "optimization" without canary data is gambling.

VII. Do Not Let Individual Cases Mislead You

One final reminder for senior engineers: failure-case-driven does not mean case-by-case optimization.

You will encounter highly specific, unusually tricky cases that are tempting to fix by adding an exception rule, inserting an if, or hard-coding a prompt snippet. Over time, however, these "point fixes" become technical debt and make the system increasingly fragile. The right approach is to examine trends. A general solution is warranted only when the same type of failure case appears at least three times. Record isolated, one-off cases in the eval set and observe them instead of rushing to change the architecture.

Avoid the opposite extreme as well: do not replace the model or framework because of one or two cases. A larger model can certainly improve scores, but it also increases cost, latency, and dependencies. In many situations, the bottleneck is not the model at all but retrieval and chunking. Exhaust the optimization headroom in the current architecture before considering a replacement.

The art of optimization lies in balancing general solutions against special-case solutions, with frequency and breadth of impact as the weights on the scale.

Summary

• Failure cases are the most valuable input for RAG optimization and reflect reality better than any benchmark. Benchmarks reveal the ceiling, while failure cases secure the floor.
• Cast a wide net when collecting them: explicit feedback + low CSAT + retry behavior + manual sampling. Always record the complete intermediate pipeline, including rewriting, retrieval, reranking, context, and raw output.
• Classification is the most critical step: recall means retrieval failed, precision means evidence was found but used incorrectly, faithfulness means hallucination, and format means formatting errors. The four categories require completely different fixes.
• Treat the root cause: tune chunking and hybrid retrieval for recall, add reranking and reduce noise for precision, use strict prompt constraints and self-checks for faithfulness, and provide few-shot examples and structured output for formatting. Do not add reranking blindly.
• Add every failure case to the eval set with expected labels, then run regression tests to prevent degradation. The eval set is the most valuable asset in a RAG system.
• Keep the loop running: collect → classify → fix → validate with evals → deploy gradually, then continue iterating. Look for trends across individual cases instead of letting outliers mislead you.

Key takeaway: the endpoint of RAG optimization is not a larger model or a longer context, but a continuously growing, labeled eval set. It is your only defense against "the system got dumber again."