RAG Evaluation Drift Control: Why Offline Gold Sets and Real Production Queries Gradually Diverge

RAG teams often maintain an offline question set and calculate recall, faithfulness, and answer scores before a release. It is essential, but it naturally ages: the product gains new features, documentation changes, users adopt different ways of asking questions, and traffic expands from internal employees to external customers. The original gold set can still score highly while production begins producing answers that miss the question. This is not degradation in a single model; it is the evaluation distribution and the real distribution gradually moving apart.

The first step in controlling drift is to treat offline samples as a dataset with a lifecycle. Each sample should record its source, creation time, product area, expected evidence, answer conditions, language, difficulty, and last validation time. Sources can include manual design, user feedback, incident postmortems, or synthetic adversarial questions, but the record must not contain only the question and the reference answer.

Select New Samples from Production Signals

You do not need to retain all raw production queries to identify problems. You can record de-identified query features, retrieval paths, zero-result events, citation clicks, follow-up questions, user corrections, human handoffs, and final satisfaction. After clustering frequent failure modes, reviewers select representative questions that can safely be added to the gold set. This lets the evaluation set evolve with real needs instead of team guesses.

Pay particular attention to "seemingly successful" samples: a user may not leave negative feedback, but immediately repeat the question, rephrase it, or move to a human channel, which may mean the first answer did not solve the problem. Combining these behaviors with retrieval evidence can distinguish missing documentation, routing errors, insufficient recall, and overly confident generation.

Do Not Treat Production Feedback as Ground Truth

A user upvote does not necessarily mean the answer is factually correct, and a user correction may be based on an outdated policy. Before adding a new gold sample, a domain owner must confirm the authoritative evidence and scope of applicability. Set expiration dates for highly time-sensitive answers and automatically revalidate them after documentation changes. Otherwise, the evaluation set will solidify old answers as "correct," in turn preventing the system from updating.

Offline evaluation should report results by strata: core safety and permission samples, mainstream task samples, long-tail questions, newly released features, and adversarial samples should each be calculated separately, rather than allowing one average to conceal a collapse in one category. For every retrieval-model or prompt change, compare differences and confidence intervals for each stratum. Limited trade-offs may be acceptable, but critical safety samples cannot be sacrificed.

Connect Offline and Production with Shadow Experiments

Run a new strategy in shadow traffic first. It must not affect user answers and should only record candidate differences from the current strategy, cost, and latency. Sample high-value queries for human comparison; when both offline and shadow results improve consistently, proceed with a small-percentage rollout. During the rollout, set automatic rollback conditions, such as an abnormal increase in missing-citation rates, zero-result rates, or human-handoff rates.

The validation process should also verify that observational data can be linked across versions: results for the same query under different retrieval strategies, document snapshots, and model versions must be comparable side by side. Without version dimensions, teams can hardly tell whether a score change was caused by content, the index, or the model.

Sampling Must Protect Both User Privacy and Evaluation Representativeness

Production samples should undergo de-identification, access control, and minimum retention before entering evaluation. Do not copy complete user queries, file names, or business data into a public evaluation set merely to improve RAG. You can retain structural features, anonymized queries, evidence types, and failure labels; when original text is truly needed, use restricted approval and set a clear retention period.

Representativeness cannot rely only on high-frequency traffic. High-value long-tail questions, low-resource languages, sessions handed off to humans after failures, and new-feature queries often have few samples but greatly affect the experience. Sample by source and risk strata, and show coverage gaps in reports. This lets the team observe real drift without turning the evaluation data itself into a new compliance risk.

Establish a Review Process for Evaluation-Set Changes

Every addition, revision, or deletion in the gold set should be reviewed like a code change. A submission includes the question source, de-identification notes, authoritative evidence, expected-answer conditions, applicable period, and risk labels; reviewers confirm that it represents a real scenario rather than an incidental case. For questions with multiple reasonable answers, record the permitted answer range and evidence that must be cited, preventing the scorer from rewarding wording while ignoring facts.

Maintain both a frozen core set and a rolling exploratory set. The core set protects release quality and changes only after strict review; the exploratory set absorbs new features and emerging production patterns and can update more quickly. By showing their scores separately in reports, teams can see whether a strategy improves stable capabilities or merely overfits the latest samples. When necessary, lock key samples by version so historical releases can be replayed and compared.

Evaluation runs should also pin model parameters, index snapshots, and scorer versions. If any of these factors changes, the results report must indicate that they cannot be directly compared, rather than plotting scores from two different environments on the same trend line. For critical releases, retain the original retrieval evidence and scoring rationale to support later investigation into "why this passed but regressed in production."

When stratified metrics become abnormal, first pause the corresponding rollout, then determine whether the cause is data drift or a scoring-pipeline failure.

Further Reading

• RAGAS Evaluation in Practice
• RAG Bad Case Mining
• Agent Online Evaluation Metrics