RAG Knowledge Freshness SLO: Do Not Monitor Indexing Success Alone

Many teams monitor whether yesterday's indexing job succeeded, but users really care about how soon a newly published policy can be answered correctly. These are different metrics: a successful job may have processed an old queue, a webhook from the source system may have been missed, and the parser may have recognized new content as empty. Knowledge freshness needs its own SLO and chain of evidence.

Defining the Start and End of Freshness

The starting point is when content actually becomes visible in the source system, not when a crawler discovers it. The endpoint is when a new, permission-filtered chunk can be retrieved by production search. The difference between the two is indexing latency. For some content, answer refresh time must also be included: if a cached answer still cites an old version, an index update does not mean the user has received the new conclusion.

Set different targets based on content risk. Product announcements or ordinary help documents can be updated within hours; security advisories, pricing, compliance rules, and incident runbooks may require minute-level targets. Do not average all sources together with one global SLA. When high-risk documents are delayed, their priority and alert severity should be raised automatically.

Change Detection Cannot Rely on Polling Alone

For source systems that support webhooks, prefer version events and retain periodic scans as a fallback for missed events. Sources without event capabilities can use content hashes, ETags, update timestamps, and sitemap diffs, but the gap between detection time and source update time must be recorded. For large document collections, compare metadata before fetching bodies to avoid the unnecessary cost of parsing everything every day.

When the same document is updated multiple times in a short period, the queue should coalesce work into the latest version rather than process every intermediate version in sequence. Otherwise, the system may spend substantial time indexing already stale content and reach the new version users actually need only at the end. Version comparison and idempotent writes are the foundation of a freshness system.

Stale Answers Need Explicit Behavior

When retrieved evidence exceeds the allowed freshness window, the system can mark the answer as potentially stale, trigger a background refresh, require the user to confirm the source time, or directly refuse to provide a conclusion in high-risk scenarios. Do not let the model answer questions such as "current price" or "latest policy" from old material without showing the source time.

Include source update time, indexing time, and version information in page and API responses. Users can see when the material behind an answer was current, and support teams can quickly determine whether the issue is indexing latency, cache latency, or a source system that was not updated. Transparent time information is more credible than a vague notice such as "knowledge base synchronized."

Driving Improvement with Operational Reporting

Regularly track freshness P50/P95, missed events, parsing failures, deletion latency, and answers marked stale by users by source, content type, and risk level. Perform root-cause analysis for sources that exceed the SLO: is the webhook unreliable, queue capacity insufficient, the parser slower, or permission review blocking progress? Every issue should have an owner and a retest date.

The value of a freshness SLO is that it turns "the knowledge base should be timely" into a measurable operational commitment. Only when the team can see the full-chain time from source-document change to user answer can RAG avoid quietly becoming an amplifier of stale information as content volume grows.

Incident Response When Delays Occur

When a high-risk source exceeds its freshness SLO, first stop presenting old material as established fact, then inspect the four stages of source events, queues, parsing, and index writes. If service cannot be restored within the target time, temporarily mark the source as "updating" and direct users to the official page or a human support channel. Transparently acknowledging that material has not yet synchronized is less risky than continuing to produce stale answers.

Afterward, add delayed samples to drills. Simulate missed webhook deliveries, queue backlogs, parsing failures, and caches that were not invalidated, then verify that alerts can actually fire before users are affected. Freshness is not a dashboard line; it is an operational capability that can still protect users under abnormal conditions.

Assign a content owner and emergency contact for every source. The indexing team can fix queues and parsers, but only the source owner can confirm whether a new policy has formally taken effect and whether the old version must be taken down immediately. Recording responsibilities in source configuration and runbooks ensures freshness alerts are handled promptly during real incidents rather than remaining on the technical team's to-do list.

Publish the sources that exceeded targets and the improvement plan in monthly reviews, turning freshness from a backend metric into a commitment jointly owned by content operations and product teams.

For sources that cannot meet targets over the long term, establish fallback strategies: reduce certainty in answers, show only original links, restrict use for high-risk questions, or temporarily remove them from the retrieval set. A freshness SLO should not force the system to pretend that material is reliable; it should help the team make safer product decisions when material is unreliable.

These strategies should be reviewed regularly and lifted once the source recovers.

Before increasing its answer weight again, use sampled retrieval and manual review to confirm that the new version covers key questions.

Further Reading

• Enterprise Knowledge Base RAG
• RAG Failure Case Mining
• RAG Tables and Structured Data