Practical RAG Evaluation: Quantifying Retrieval Quality with RAGAS

The most frustrating part of building RAG is this: you switch to a different chunking strategy, the answers seem to get "better," but you cannot say whether they really did - and nobody notices when someone switches back next time. RAG optimization without evaluation is entirely driven by gut feel. RAGAS is built to address this: it splits "retrieval quality" and "generation quality" into several measurable metrics, so every change has an objective score.

1. First, Be Clear About What You Are Evaluating

A RAG system has two independent quality stages from query to answer:

• Retrieval stage: Did it retrieve the content that should have been retrieved? Of what it retrieved, how much is relevant?
• Generation stage: Is the answer faithful to the retrieved content (no hallucinations)? Does it directly answer the question?

The essence of RAGAS is scoring these two stages separately. Only then can you tell whether a poor answer comes from retrieval failing to find the material (fix chunking/recall) or the model failing to properly use the material it found (fix the prompt/model).

2. Four Core Metrics

RAGAS uses four metrics, each measuring one thing:

Diagnostic logic:

• Low Context Recall → Retrieval did not find everything. Improve chunking (finer/more appropriate), improve query rewriting, or add hybrid retrieval.
• Low Context Precision → A pile of irrelevant content was retrieved. Add reranking, reduce top-k, or change the embedding.
• Low Faithfulness → The model is making things up instead of using the context. Strengthen the prompt ("Answer only based on the context; if it cannot be found, say you do not know."), or use a model that follows instructions better.
• Low Answer Relevancy → The answer misses the question. Improve the prompt (make the question scope explicit), and check whether query rewriting went off track.

Check Context Recall first (whether relevant content can be retrieved), then Faithfulness (whether the model is making things up) - these are the two most important foundations. Tune Precision and Relevancy during the optimization phase.

3. Building an Evaluation Set: The Hardest but Most Valuable Step

RAGAS needs an evaluation set, with each entry containing at least:

{
  "question": "MCP 协议默认的传输方式有哪些？",
  "ground_truth": "stdio 和 HTTP（Streamable HTTP/SSE）两种。",
  "contexts": ["...（可选，不提供则由 RAGAS 从你的检索结果填）"],
  "answer": "（可选，不提供则由评估用的 LLM 生成）"
}

Three sources for building the set:

1. Real user questions (best) - Pull actual user wording from logs or support channels; it is closest to the real distribution.
2. Manually devised questions - Cover typical scenarios, boundaries, and common failure points.
3. LLM-assisted generation - Feed documentation to a model and ask it to "create questions for this passage and provide reference answers," then review the results manually.

Scale: 50-200 items is enough to get started. With fewer, results are not statistically significant; with more, labeling costs rise. Coverage matters more than quantity - cover simple, complex, multi-hop, ambiguous, and boundary questions.

ground_truth does not need to be the exact reference-answer wording. The key is the decision criteria: which key facts the answer should include.

4. Get It Running

from datasets import Dataset
from ragas import evaluate
from ragas.metrics import (
    context_precision, context_recall,
    faithfulness, answer_relevancy,
)

# data: 你的评测集,每条跑过 RAG 后补上 contexts 和 answer
dataset = Dataset.from_list([...])

result = evaluate(
    dataset,
    metrics=[context_precision, context_recall, faithfulness, answer_relevancy],
)
print(result)  # 各指标的平均分

RAGAS automatically scores with an "LLM as a judge" (you need to provide an LLM), avoiding manual review item by item. Note:

• The judge model must be strong enough. Scores from a weak judge model are unreliable. Usually, use a judge model stronger than the system being evaluated.
• Faithfulness judgments depend most on judge capability. If the judge model is weak, this score becomes unstable.
• A run costs tokens (N 条 × 多次裁判调用), so do not start with evaluation sets containing thousands of items.

5. Use Results to Locate Problems

After obtaining the four scores, do not look only at the total. Slice bad cases by dimension:

• Slice by question type: Is recall especially low for multi-hop questions? → Optimize multi-hop retrieval separately.
• Slice by document source: Does one document type have low recall? → That document type has a chunking/embedding problem.
• Inspect the specific content of low-scoring cases: for low faithfulness, see what the model is fabricating; for low recall, see which passage was missed.

The evaluation set plus bad-case analysis is the real optimization engine. Scores tell you "where it is failing," while bad cases tell you "why it is failing."

6. Put Eval into the Development Loop

The ultimate use of RAGAS is to put it into CI and run it on a schedule:

• Change a prompt, switch embeddings, or tune chunking → run the evaluation → block the change when scores fall (or at least record it).
• Run it regularly to monitor production regressions (model API upgrades and documentation-base changes can cause score drift).
• Every time you find a production bad case, add it to the evaluation set → prevent the same type of problem from regressing again.

This makes the evaluation set increasingly representative and the system increasingly stable over time - that is the real long-term value of RAGAS.

7. Common Misconceptions

1. Looking only at answer relevancy. It is the easiest metric to "look high" (the model answers fluently), but it hides retrieval and faithfulness problems. Look at all four metrics together.
2. Using the same judge model and model under test. It is grading itself, so scores become inflated. Use a stronger independent model as the judge.
3. Keeping the evaluation set unchanged. As the business and documentation base change, an old evaluation set loses representativeness. Keep updating it.
4. Chasing a score of 100. Moving from 0.7 to 0.85 is easy; moving from 0.95 to 0.99 is extremely difficult and has diminishing returns. Consider ROI and stop when it is good enough.

Summary

You only need one test of RAG-optimization maturity: after a change, can you say "recall rose from what to what," instead of "it feels better"? RAGAS helps turn gut feel into numbers.

The four metrics (context precision/recall, faithfulness, and answer relevancy) each cover one stage. Build an evaluation set with 50-200 items, run it, and slice bad cases by dimension. Once this loop works smoothly, your RAG moves from "mystical debugging" into "engineering optimization." The upfront investment (building the set and integrating eval) is somewhat heavy, but once it is in place, every subsequent change benefits - it is the highest-ROI infrastructure in a RAG project.