Zero-Downtime Rebuilding of Vector Indexes

Anyone who works with RAG long enough will eventually reach this point: you want to change the embedding model, revise the chunking strategy, or update a large amount of historical data. These operations have one thing in common: they all require rebuilding the vector index. In production, the two biggest risks of rebuilding an index are downtime and discovering after the switch that retrieval quality has degraded.

This article explains several zero-downtime rebuild strategies in depth.

1. Why Vector Indexes Need to Be Rebuilt

First, let's clarify which scenarios make rebuilding unavoidable:

1. Changing the embedding model. The old model's retrieval quality is no longer adequate, so you want to switch to BGE-M3 or a newer OpenAI model. The new model's vector dimensions and space are incompatible with the old model, making the old vectors completely invalid. The entire dataset must be recomputed.
2. Changing the chunking strategy. Switching from fixed 512-token chunks to semantic chunking changes the boundaries and content of every chunk, so the corresponding vectors must also be recomputed.
3. Major data changes. Examples include restructuring the knowledge base, performing bulk deduplication, or reingesting data after metadata cleanup.

Why not take the service offline during the rebuild? Because a rebuild does not take just a few seconds. Re-embedding and indexing millions of documents can take tens of minutes or several hours. During that time, either every query returns 503, or queries hit dirty data that is "half new and half old," producing inconsistent results. A production RAG system cannot operate that way.

The core conflict is therefore: building the new index takes time, but the production service must remain available throughout the process. The following strategies all address this conflict.

2. Strategy One: Blue-Green Indexes with an Atomic Cutover

The classic approach is borrowed from blue-green deployments. Maintain two indexes simultaneously: one serving production traffic (blue) and one being built (green).

[Query router] ──> Blue index (v1, production)
                  Green index (v2, full rebuild running in background)

       ...Rebuild complete, validation passed...

[Query router] ──atomic cutover──> Green index (v2)   Blue index retired

Process:

1. Create a new collection or namespace, such as docs_v2. Run a full rebuild in the background, reprocessing every document with the new embedding model or chunking strategy.
2. During the rebuild, queries and writes continue to use v1. The green index remains invisible to users.
3. Once the green index is ready and has passed validation, switch the query router to v2 in a single operation by changing a configuration value or rollout flag.
4. Monitor the new index and retire v1 after confirming that it is stable.

Advantages: The cutover is atomic, so users never encounter an intermediate state. Rollback is also simple: switch back to v1, which has not yet been deleted.

Pitfall: New writes during the rebuild went only to v1, not v2. Before the cutover, you must apply the incremental changes by synchronizing documents created or modified in v1 during the rebuild to v2. Otherwise, data will be lost. A common approach is to pause writes for a few minutes before the cutover and apply the remaining changes, or use CDC or a change log to catch up.

This strategy suits large datasets, workloads that can tolerate a brief pause in writes, and teams that want a clean cutover.

3. Strategy Two: Dual Writes for a Smooth Transition

Do not want an abrupt cutover? Have both indexes serve traffic simultaneously and transition gradually.

       Writes ──┬──> v1 index (old embedding)
                └──> v2 index (new embedding)

       Reads ──> v1 (transition period) ──validate that v2 quality is comparable──> v2

Process:

1. Deploy dual-write logic so every document write or update goes to both v1 and v2, using the corresponding embedding model for each.
2. Backfill all historical data into v2 in the background.
3. After the backfill finishes, v1 and v2 contain consistent data because dual writes kept incremental changes synchronized.
4. Gradually shift read traffic from v1 to v2 (5% → 50% → 100%).
5. After all traffic has moved to v2, stop dual-writing to v1 and retire it.

Advantages: No downtime, no abrupt transition, and read traffic can be shifted gradually, providing stronger quality assurance.

Pitfalls:

• Write costs double because every operation requires two embedding computations and two writes.
• Code complexity increases because you must maintain dual-write logic and consistency. What happens if writing to v1 succeeds but writing to v2 fails? You need a compensation mechanism.
• The transition period may last a long time.

This strategy suits services that cannot pause writes, require a gradual rollout, and have strict consistency requirements, such as consumer-facing search products.

4. Strategy Three: Shadow Traffic for Validation Before Cutover

If you are unsure about the quality of the new index because a different embedding model or chunking strategy could improve or degrade retrieval, do not send user traffic to it immediately. First run shadow queries against the new index, compare the results with production, and switch only after validation succeeds.

User query ──> v1 index ──> Return results to the user (production)
           └─(async)─> v2 index ──> Write results to logs without returning them

            Compare v1 vs v2 offline using Top-K overlap / click-through rate / manual sampling

Process:

1. Complete the full rebuild of v2 using either blue-green indexes or dual writes.
2. Send production queries to v1 synchronously, returning its results to users, and to v2 asynchronously, logging its results without returning them.
3. Compare both result sets offline using Top-K overlap, retrieval quality measured with an evaluation set, and response latency.
4. If v2 matches or exceeds v1 in quality, switch traffic. If quality degrades, tune it first and do not switch.

Value: This is the only way to observe how the new index performs against the real production query distribution before switching traffic. Even the most comprehensive evaluation set cannot match real traffic.

The cost is the additional compute required for asynchronous queries, but it is worth paying compared with the cost of switching first and then discovering that retrieval has collapsed and production is failing.

5. Incremental Updates: A Full Rebuild Is Usually Unnecessary

Stop and ask: Do you really need a full rebuild?

Many teams reflexively recompute everything even though only a small portion of the data changed. For example:

• Only a few hundred knowledge base documents were updated → Recompute vectors only for those documents.
• The chunking strategy did not change and only metadata was added → The index does not need to change at all.
• Only a batch of old documents has formatting errors and needs to be rechunked → Rechunk only that batch.

A full rebuild is the last resort, not the default. Design the RAG system for incremental updates from the beginning:

• Give every document or chunk a version number and an embedding model identifier, such as embed_model: bge-m3 and embed_version: 2.
• Query only the current version. During an upgrade, batch-recompute documents that lack the new identifier as needed, progressing in small steps.
• Always process new writes with the new version while historical data gradually catches up.

Breaking a "rebuild" into a continuous series of small-batch migrations avoids the risks of a single large operation. A full rebuild is unavoidable only when the embedding dimensions change or the chunking strategy is completely restructured.

6. Special Considerations When Changing Embedding Models

Changing the embedding model is the most demanding type of rebuild because once the dimensions change, the vector spaces become completely incompatible. Incremental updates and mixed usage are impossible: combining old and new vectors in one index is incorrect.

Several practical reminders:

• A dimension change requires a full rebuild; there is no shortcut. Choose an embedding model carefully and pin its version, as discussed in the previous article on embedding selection. Changing the embedding model can cost more than changing the large language model.
• Do not "mix queries" during the transition. You cannot encode the same query with the old embedding model at one moment and the new one at another. If the index contains vectors from the new embedding model while the query uses the old one, every similarity score will be wrong. The embedding models used for queries and the index must match exactly. This is an inviolable rule.
• If the change is unavoidable, prefer blue-green indexes or dual writes. Treat the entire index as an indivisible unit and never mix old and new vectors.
• One convenient but effective strategy is: if the new embedding model has higher dimensions and better quality, the rebuild may be worth the cost, but time it carefully by using an off-peak period, preparing rollback, and testing with shadow traffic.

7. Engineering Essentials: Cost, Rollback, and Monitoring

Whichever strategy you choose, you must address these three areas:

1. Cost. A full rebuild means re-embedding every document in the database, and embedding API calls or GPU inference are not cheap. Recomputing millions of documents can cost hundreds to thousands of yuan with a commercial API. Estimate the token volume in advance, run the job during off-peak hours, and rate-limit it in batches if necessary. Do not let it overwhelm either your bill or production latency.

2. Rollback. Before switching, decide how to recover if something goes wrong:

• Blue-green: Keep the old index for N days and switch back if problems arise.
• Dual writes: Roll out read traffic gradually and immediately reduce the rollout to 0% if problems appear.
• Shadow traffic is itself a rollback safety net: if you have not switched, there is nothing to roll back.
• Do not rush to delete the old or new index. Observe at least one week of stable operation before retiring anything, so you retain a recovery option.

3. Monitor quality before and after the cutover. The cutover is not the finish line; monitoring is. Watch these signals:

• Retrieval metrics: Add production instrumentation and compare Top-K click-through rate, zero-result rate, and repeat-search rate before and after the cutover.
• Evaluation-set regression: Run a fixed evaluation set every day and check whether the curve drops on the cutover date.
• Latency and error rate: Verify that query latency and timeout rates for the new index remain normal.
• Offline comparison: Review overlap and manually sampled accuracy in the shadow-traffic logs.

Have someone monitor the system on the day of the cutover. Roll back immediately if a problem appears instead of trying to push through it.

Summary

Rebuilding vector indexes is one of the riskiest RAG operations tasks, but the principles are clear:

1. First determine whether a full rebuild is truly necessary. Incremental updates are sufficient most of the time; a full rebuild is the last resort.
2. Blue-green indexes suit scenarios that can briefly pause writes, offering an atomic cutover and simple rollback.
3. Dual writes suit production services that cannot pause writes, providing a gradual rollout at the cost of doubling write operations.
4. Shadow traffic is the safety net before switching traffic, validating how the new index actually performs against production queries.
5. Changing embedding models is the most demanding case: a dimension change requires a full rebuild, query and index embedding models must match exactly, and old and new vectors cannot be mixed.

Remember: Rebuilding an index is not frightening; switching traffic without a rollback path is. Prepare the strategy, validation, rollback, and monitoring, and a rebuild becomes a boring routine operation. In production systems, boring is the best possible state.