Data Backfill Generation Cutover: Frozen Ranges, Dual-Compute Reconciliation, and Atomic Reader Migration

A tax-rate bug affected the last twelve months of orders. The team fixed the SQL and re-ran the production table day by day; the real-time job wrote today's partition at the same time, the two executors disagreed on yesterday's boundary, and a querier first saw three months of new logic and nine months of old logic. The tasks eventually all went green, yet no one could say which generation the deleted rows, late orders, and the rerun's new source revision landed in.

Airflow backfill can create runs for historical logical dates, and a dbt incremental model only processes the new or changed rows its config selects; these mechanisms handle scheduling and building, and don't automatically provide atomic visibility across all partitions. Database migrations handle table structure, operational CSV import handles promoting one file; this article targets continuously changing data sources and large-scale historical recomputation.

Backfill first becomes an immutable release

Publish dataBackfillRelease.v1, binding target dataset, old/new transform digest, source snapshot or per-partition watermark, UTC half-open range [from, to), partition spec, schema, delete semantics, target generation, concurrency and resource budget, expected partition/count/root, owner, approval and deadline. Operators who change the range or SQL create a new release; they can't reuse an old run ID.

schema: dataBackfillRelease.v1
release: order-tax-fix@3
range: { from: 2025-07-01T00:00:00Z, to: 2026-07-01T00:00:00Z }
sourceSnapshot: orders:snapshot-8841
transformDigest: sha256:7b2...
targetGeneration: tax-fix-g3
writeMode: REPLACE_RANGE

Range boundaries are interpreted in the target business time and the timezone is frozen. The absence of REPLACE_RANGE means deletion, and must be generated from a complete source snapshot; when incremental events are incomplete you can't infer "the old row should disappear." When the whole source can't be frozen, each partition saves a start watermark, real-time events write to a change buffer at the same time, and after backfill completes they're caught up by stable event ID and source revision. Reading the current database as you run lets each partition of the same release be based on a different world.

The catch-up watermark can't rely on "reading the offset one last time at the end." The write ingress holds an ingressEpoch, and every target commit must compare the current epoch in the registry. When ready to cut over, sealIngress competes with the old writer for the same CAS: the winner pushes epoch E from OPEN to SEALED and records the per-partition barrier of confirmed commits; an old commit that loses to the seal must not land. Upstream ingestion still flows into the persistent log, but events after the barrier are buffered first; only after the candidate generation covers all events <= barrier and recomputes the root is the final cutover allowed.

An independent generation avoids half-visible state

The state machine is DRAFT -> SNAPSHOT_PINNED -> BUILDING -> INGRESS_SEALED -> CAUGHT_UP -> RECONCILED -> APPROVED -> ACTIVE -> RETIRED. A worker only writes (dataset, targetGeneration, partition, businessKey), and each partition receipt binds the input root, output root, transform, and attempt; retries can only overwrite the same digest. The production reader can't see the new generation at all before cutover.

Reconciliation simultaneously verifies partition closure, row-count buckets, the business-key set, conserved quantities like amounts, null/deletion distribution, and the expected diff between old and new transforms. Comparing only total row counts would miss one added and one removed. Training dataset snapshots can serve as source evidence, atomic training-shard publishing provides the set root; real-time catch-up reuses the offset of database CDC, but cutover is still decided by the backfill registry.

The single acceptance point is the dataset registry using CAS to compare the expected old generation, the sealed ingress epoch, the exact barrier, the complete partition manifest, the catch-up root, and the approval summary, while simultaneously switching the active pointer and publishing a new ingress epoch. Afterward only persistent events after the barrier are routed to the new generation; replaying the same event ID/source revision doesn't duplicate. A query fixes the generation at request start and can't switch generations across pages. The old generation is retained until the maximum reader lease and rollback window end, and late old-worker commits fail.

Manufacturing breakage at boundaries and reruns

The fault suite covers the to boundary being included by both batches simultaneously, the source snapshot going stale, two workers on one partition, an old writer committing before and after the seal CAS, post-barrier events arriving before the active CAS, build completion with a lost receipt, missing deletion events, real-time buffer overflow, an old worker waking up after cutover, a reader long transaction crossing generations, an object write succeeding but the manifest not committed, and new data continuing to arrive during rollback. The expectation is retrying the same generation, blocking cutover, or an explicit RECONCILIATION_REQUIRED—never patching a few rows in place and marking success.

The release gate can require: 100% planned partition coverage with 0 overlap; 100% recomputability of input/output manifest roots; 0 difference in business conserved quantities; 0 unexplained diff; 0 lost or duplicated events in seal-vs-cutover races; 0 canonical writes to the old generation after cutover; 0 mixed generations across 10,000 concurrent reads during cutover; rollback drill RTO under 15 minutes. The resource gate also caps source QPS, warehouse cost, and real-time lag—you can't drag down the current business to backfill history.

On rollout, first do a full dry-build in an isolated namespace, then let internal reports read in shadow and compare; then cut over one low-risk date range, and finally expand. On anomaly, CAS the active pointer back to the still-retained old generation, while replaying new-period events to the old generation or entering explicit reconciliation—you can't just switch the read pointer and lose writes after cutover. Bulk customer operations control external customer side effects; data backfill only corrects data projections and must not re-send emails or refunds by re-running SQL.