RAG Incremental Indexing Consistency: A Production Design for Updates, Deletes, and Duplicate Events

RAG indexing incidents are often not cases of “no data at all,” but harder-to-detect partially consistent states: the source has been updated but old chunks are still retrieved, document permissions have been withdrawn but vectors remain searchable, or webhook replay causes the same passage to appear three times. The answer sounds certain to the user, yet it is based on stale knowledge. These problems cannot be solved with periodic full rebuilds; the document lifecycle must be treated as a traceable data pipeline.

Version Both Documents and Chunks

Every source document needs a stable documentId, content version, source update timestamp, and deletion status. Each chunk created after splitting should also carry documentId, documentVersion, a chunk index, and a content hash. Retrieval should allow only the current version. Older versions may be retained for a period for rollback and auditing, but they must not participate in retrieval alongside the new version.

Do not treat a filename or URL as a unique identity. Knowledge bases often move pages, rename files, or migrate systems; a path change should not automatically create a new document. A stable ID can come from the source system's primary key. If there is no primary key, establish URL canonicalization and content fingerprinting policies, along with an explicit manual merge workflow.

With At-Least-Once Events, Writes Must Be Idempotent

Most webhooks and queues may deliver duplicate events. When an indexing worker receives a “document update,” it first compares the version and hash: an older-version event is discarded directly, an event with the same version and hash returns as already processed, and only a higher version proceeds to parsing and embedding. When writing to the vector database, use deterministic chunk IDs composed of documentId + version + chunkIndex rather than generating random IDs each time.

This prevents duplicate vectors even when a worker restarts, a queue is replayed, or someone manually clicks retry. Failure records retain the source event ID, document version, failure stage, and retryable flag. A retry does not fetch everything again; it resumes from the failed stage and confirms that the current version has not already been superseded by a later event.

Deletion Is Not Just Hiding It in the UI

Deletion events should be handled with priority. For indexes that support physical deletion, delete all current chunks for that document; for scenarios that require retaining an audit trail, write a tombstone and have retrieval filter deletedAt is null. In either case, caches, full-text indexes, vector indexes, and citation pages must be invalidated together. Deleting only vectors while leaving caches intact can still let users read old answers for several minutes.

Permission revocation can also be treated as a logical deletion. Do not wait for a nightly batch job to remove sensitive documents; permission events should trigger immediate filtering or index updates. High-risk data should primarily rely on retrieval-time ACL filtering, with asynchronous cleanup used to reduce storage exposure, rather than trusting asynchronous deletion to complete in time.

Use Reconciliation to Prove the Data Is Actually Consistent

Each day or hour, sample and compare the source system with the index: current document count, version distribution, missing chunks, orphan chunks, deletion latency, and failure queue length. Reconciliation results should identify the relevant documentId, not merely provide a total. When releasing a new parser or chunking strategy, first rebuild in a shadow index and compare retrieval and coverage before switching the production alias.

The goal of incremental indexing is not to put new documents into the vector database as quickly as possible. It is to ensure that, at any time, the system can explain “which version this answer came from and why it is still visible.” With versioning, idempotency, and reconciliation, RAG can operate reliably over the long term like a dependable data system.

Migrations and Rollbacks Must Also Be Part of Index Design

When the chunker, embedding model, or metadata schema changes, do not overwrite the original index in place. Create a new index version, validate document coverage, permission filtering, retrieval quality, and latency against fixed samples, then switch the query alias to the new version. Retain the old index until the rollback window ends, so a parsing error does not leave you waiting for another full build.

Rollback does not mean rolling source document versions backward; it means restoring the index implementation used by queries. Record the configuration, model, document version range, and validation report for every index switch. When an incident occurs, you can quickly determine whether the difference was caused by source content, parsing, embedding, or retrieval configuration. This gives every step of the data pipeline a recoverable boundary instead of making it one irreversible batch process.

Index worker monitoring should include event dwell time, current-version lag, failure rate for each source, and orphan chunk count. Break these metrics down by source and document type to discover when a PDF parser or knowledge-base connector is systematically falling behind. After identifying an issue, first prevent an old version from being marked as current, then perform compensating replay.

Further Reading

• Enterprise Knowledge Base RAG
• RAG Bad Case Mining
• Advanced pgvector