RAG Embedding Migration: Dual Indexes, Version Cutovers, and Rollbackable Vector Upgrades

Upgrading an embedding model often brings higher dimensionality, better multilingual capabilities, or lower cost, but it also changes the vector space. Old query vectors cannot be mixed with new document vectors, and changes in chunking strategy can invalidate historical citations. Treating migration as an ordinary index rebuild makes it easy to discover only after the cutover that recall declined, latency spiked, or some documents have no new vectors at all.

The first step is to explicitly version vectors and chunking strategies. Each document chunk records embeddingModel, embeddingVersion, chunkingVersion, sourceHash, createdAt, and indexing status. Queries also record the version used. Without version fields, it is impossible to explain why the same question returns entirely different results at two points in time.

Avoiding a One-Shot Bet with Dual Indexes

During migration, retain old index A and new index B. New writes should first be dual-written according to capacity policy, while historical data is asynchronously backfilled into B. Before backfill is complete, query results must make missing coverage identifiable instead of pretending B is complete. Shadow traffic runs A/B retrieval concurrently and records candidate overlap, relevance labels, latency, and no-result rate, while answers continue to use the old index by default.

Cutover criteria should not be limited to "100% backfilled." They should at least include no degradation in offline evaluation for critical domains, no material increase in zero-result rates for online shadow queries, consistent permission filtering, tail latency within budget, and citation pages that still locate the correct original text. For multi-tenant systems, phase the cutover by tenant or dataset to prevent one anomalous index from affecting all customers.

Changes in Chunking Need Separate Validation

Many migrations appear to be model upgrades, but their actual effects primarily come from chunk size, overlap, title injection, and table handling. Keeping the chunking version separate from the embedding version makes it possible to determine where an improvement came from. During migration, select long documents, tables, code, short FAQs, multilingual documents, and frequently updated documents as samples, then verify that chunks retain readable semantics and citations cover the complete answer.

If new chunks concatenate a sensitive document with unrelated context, permission filtering may still be correct, but answers will become difficult to explain. Retrieval-quality evaluation should include relevance, contextual completeness, citation precision, and correct abstention, rather than just top-k similarity.

Rollback Must Be an Executable Condition

Define rollback thresholds in advance, such as a drop of more than 3% in critical-query-set accuracy, a doubling of the no-result rate, P95 latency exceeding the budget, more than zero permission anomalies, or a failed new-index health check. When triggered, switch the read path back to A and stop rolling out B, while retaining diagnostic data. Do not debate whether to roll back during an incident; that allows model and business risk to grow together.

After a rollback, distinguish between index-service failures, missing backfill data, model semantic changes, and chunking problems. Different causes determine whether the next experiment should adjust retrieval parameters, repair the data pipeline, or replace the model. Retain a configuration snapshot and evaluation report for every migration so later comparisons are evidence-based rather than based on memory.

Backfill Jobs Also Need Resource and Consistency Controls

Backfilling historical documents is often more likely to cause incidents than the model calls themselves. It reads large numbers of source files, consumes queue capacity, writes to the index, and triggers cache invalidation. Partition work by tenant, dataset, and time window, and give every partition progress tracking, failure retries, maximum concurrency, and validation hashes. Do not count only "processed records"; also compare source-document counts, successful-write counts, skip reasons, and the number of new-version documents that can be queried.

During dual writes, update and delete events must affect both A and B. Otherwise, the new index may already lag behind the actual data by the time backfill completes. Retain the last event sequence number or version stamp for every source document, and mark work complete only when the index version is no earlier than the source version. Launch rehearsals should include mid-backfill updates, deletions, and worker restarts, confirming that both indexes eventually converge and do not write deleted content back.

Query Parameters Must Also Be Frozen by Version

New and old embeddings have different similarity distributions, so distance thresholds, top-k values, and reranking weights that worked for the old index should not be copied unchanged. Save a query-parameter snapshot for each index version and calibrate it separately on the offline set. Otherwise, the team may mistake a recall drop after a model upgrade for an index failure when it is actually still using unsuitable thresholds.

After the cutover, retain dual-read diagnostics for a period, especially for low-scoring results that are selected after reranking. If the new space requires a higher candidate count to achieve the same result, include the cost change in the release decision. The model, chunking, index, and parameters together form a retrieval version; without any one of them, reliable rollback is impossible.

The acceptance report should list coverage, quality, latency, and cost together to avoid approving a release based on a single score.

If any item crosses its preset threshold, retain the old read path and enter diagnostics rather than cutting over with a known issue.

Further Reading

• Vector Index Rebuild
• Embedding Model Selection
• RAG Incremental Indexing Consistency