RAG Evidence Conflict Resolution: How to Stop Fabricating Conclusions When Retrieval Results Contradict

“Having citations” does not mean “the answer is correct.” A single question may retrieve an old version of an operations manual, a new version of product documentation, policy copies from different tenants, and a discussion record with no publication date. If the model mixes them into a fluent conclusion, the user sees only a seemingly confident but incorrect answer.

Evidence conflicts should be explicitly identified before generation. In addition to its body text, every retrieved fragment needs a source type, version, publication date, applicability scope, permission labels, author or publisher, content hash, and revocation status. Without this metadata, the system cannot reliably distinguish a “historical record” from a “current rule.”

Rank Trustworthiness Before Deciding Whether Sources Can Be Merged

Ranking should not rely on vector similarity alone. Typical priority should include: authoritative configuration for the current tenant or project over general documentation; published specifications with explicit versions over unverified discussions; newer versions when they explicitly supersede older versions; and materials that match the user's permissions and the question's scope. This order must vary by domain. Legal, medical, and financial content, for example, also requires stricter source governance.

Even if two sources are both trustworthy, that does not mean they can be merged. If one says a quota is 100 and another says it is 500, the system must determine whether they refer to different plans, times, or regions. When it cannot resolve the difference, the answer should show the conflict and the clarification needed rather than implicitly choosing the larger number.

Detect Conflicts with Structured Claims

For critical facts, convert fragments into comparable claims: subject, predicate, object, unit, time range, and applicability scope. For example, plan=pro, max_requests=1000/day, effective_from=2026-07-01. A conflict candidate can be flagged when the subject and predicate are the same, the time and scope overlap, and the objects differ. A model can help extract candidates, but the final comparison logic should be testable.

claim A: project=x, retention=30d, source=policy-v3
claim B: project=x, retention=90d, source=policy-v4
resolution: B supersedes A after 2026-06-01

For unstructured technical explanations, there is no need to force a knowledge graph, but at minimum retain the “set of fragments supporting a conclusion.” The generator may answer only from the same compatible evidence set; if the evidence contains unresolved conflicts, it should switch to clarification or citation-based summary mode.

Responses Should Reflect Uncertainty

When a conflict can be resolved by time or scope, the answer should state the conditions, for example: “For projects created after 2026-07, the current policy is 30 days; older projects may still be subject to 90 days.” When it cannot be resolved, clearly tell the user that two inconsistent sources were found, and provide the sources, dates, and next verification method. Such an answer may not look definitive, but it is more valuable than fabricated certainty.

For Agents that trigger write operations, unresolved conflicts must block execution. For example, if two sources disagree on a deletion retention period, the system should require human confirmation or query the authoritative configuration rather than choose the more aggressive action. Elevating “insufficient evidence” into a state rather than a prompt instruction helps prevent high-risk automation.

Calibrate the Policy with Replay Samples

Build an evaluation set containing version supersession, tenant differences, missing publication dates, and real contradictions. After every adjustment to ranking or the model, check whether the system selects the correct authoritative source, incorrectly merges conflicts, or successfully clarifies when it cannot determine the answer. In production, also monitor conflict rate, user follow-up rate, and citation click-through rate; if a certain type of data frequently conflicts, the issue may lie in upstream content governance rather than RAG itself.

Upstream Content Revocation Should Immediately Affect Answer Eligibility

Conflict handling cannot be separated from the content lifecycle. After a document is revoked, a policy is superseded, an error is discovered in a data source, or a permission scope changes, the relevant fragments must be removed from the answerable set rather than merely receiving a slight similarity penalty. The indexing pipeline needs to consume revocation events, update content status, and invalidate caches and citation pages at the same time. For distributed indexes that cannot be deleted immediately, the query layer should maintain a revocation list as a final safeguard.

During validation, deliberately revoke a document that was frequently cited, then check whether old answers still cite it, whether the cache is cleared, and whether conflict detection surfaces the new version. If the system stops using old evidence only after the next full rebuild, content governance still has a dangerous window. Treating revocation, supersession, and permission changes as first-class retrieval events is necessary for citations to remain current.

Conflict Escalation Requires a Clear Owner

When the system cannot automatically adjudicate a conflict, it cannot simply leave the issue in logs. Route the conflict to a domain-specific review queue, including sources, applicability scope, detection rules, and recommended options. The review conclusion should update authoritative content or rules rather than merely close a one-time alert; otherwise, the same contradiction will recur in the next query.

Further Reading

• RAG Citation Provenance
• RAG Document Freshness SLO
• RAG Metadata Filtering