Lakehouse Snapshot Retention and Recovery Drills: Reference Watermarks, File Reclamation, and Legal Hold

When a data team sees object storage usage growing, it runs expire snapshots and remove orphan files. The SQL succeeds and storage usage falls, but only the next day does the team discover that a training job still references yesterday's snapshot ID, an audit export still needs a manifest from a month ago, and files produced by a long-running write job were removed as orphans because they "appeared unreferenced." Atomic commits in snapshot-based tables protect only a single metadata switch; they do not decide which history still has active readers for the team.

Apache Iceberg represents table state through snapshots, manifest lists, manifests, and data/delete files. Maintenance can expire historical snapshots and clean up old metadata and orphan files. The danger is that reclamation is irreversible: a late reader, an unregistered branch, or a still-running writer can make "not referenced by the current table" a false conclusion. Training dataset snapshots define dataset lineage, while signed RAG index artifacts protect derived indexes. This article governs when underlying table files may actually be deleted.

The retention policy must account for every reference

Publish lakehouseRetentionPolicy.v1, binding the table UUID, format/spec version, catalog, protected branches/tags, minimum snapshot age, reader lease limit, maximum writer duration, disaster-recovery points, legal hold generation, object versioning policy, and reclamation implementation digest. Before starting, every training, backfill, export, or audit job registers a snapshotLease containing the snapshot ID, purpose, owner, expiresAt, and reader generation. Unregistered readers must not assume that time travel will remain available later.

schema: lakehouseRetentionPolicy.v1
table: orders/7f31...
protectedRefs: [main, audit-q2]
minSnapshotAge: 30d
maxReaderLease: 14d
maxWriterDuration: 6h
legalHoldGeneration: 19
orphanSafetyInterval: 12h

The candidate reclamation set may only be computed from a fixed catalog generation. The union of live refs, unexpired reader leases, staging prefixes for in-progress writers, disaster-recovery checkpoints, and hold scopes defines the non-deletable closure. orphanSafetyInterval must exceed the longest writer visibility delay and be supported by measurements; relying only on file mtime will misclassify slow uploads, cross-region replication, or clock drift. A deletion request carries the candidate manifest root instead of expanding its scope through a fresh background enumeration.

Split deletion into reversible marking and irreversible purging

Split the state machine first: objects covered by a live ref, valid lease, writer, disaster-recovery point, or hold enter PROTECTED and leave the candidate set. Only DISCOVERED -> UNREFERENCED_ELIGIBLE -> QUARANTINED -> RESTORE_VERIFIED -> DELETE_APPROVED -> DELETED -> VERIFIED may proceed. Before an expired snapshot is removed from the default history available to new readers, create a quarantine/<candidateId> tag used only for isolated recovery, or archive a signed metadata bundle, ref root, and all object version IDs sufficient for an independent bootstrap. This temporary tag pins the files but does not alter the eligibility evidence the candidate already satisfied. During quarantine, a hold or new external reference still sends the candidate back to PROTECTED.

The only irreversible acceptance point occurs when the deleter uses CAS to compare the table generation, protected-ref root, lease watermark, hold generation, and verified quarantine tag/bundle root, then archives the exact object inventory, removes the temporary tag, and submits the inventory to the storage layer. Any change sends the candidate back for recomputation. Do not first run an expiration operation that removes the last snapshot metadata chain and then assume that data files not yet physically deleted are sufficient to restore the table.

Object-deletion receipts record the version ID, etag/digest, request and confirmation times, and provider result. For a versioned bucket, they must also document how delete markers and old versions are handled. Readable metadata JSON does not imply complete data. The restorer should bootstrap the catalog from the quarantine tag or independent bundle, then recursively verify the digests of snapshots, manifests, and data/delete files. Atomic publication of training shards provides an approach to set integrity, while database migrations remind us that old readers and new formats need a compatibility window.

Prove recoverability before every reclamation

The fault-injection matrix covers control-plane delays after reader registration, branch-name case ambiguity, writer uploads without commits, eventual consistency in object listings, a hold racing with delete CAS, a protected snapshot mistakenly entering the candidate set, a missing metadata-bundle layer after removal of the default ref, a model still referencing an expired snapshot, lagging cross-region replicas, lost deletion receipts, and version-marker recovery. The expected result is delayed reclamation or an explicit DELETE_UNKNOWN; a successful current query must not substitute for historical recovery.

Gates can require: zero deleted objects among protected refs and valid leases; zero difference between the candidate manifest and actual deletion set; all 30 randomly selected weekly snapshot recoveries from quarantine passing root verification; quarterly full recovery of critical tables from cold backup with an RTO under four hours and an RPO equal to the declared checkpoint; zero new deletion acceptances after a hold takes effect; and 100% of UNKNOWN deletions entering reconciliation. Storage savings cannot compensate for even one unrecoverable loss.

Start rollout by generating candidate reports only and comparing them with existing maintenance jobs for two weeks. Then allow the new policy to process one test table containing no regulated data before expanding by table cohort. When anomalies occur, stop the irreversible deleter, restore the default ref from a protected tag or verified bundle, and extend the quarantine period. Objects already deleted can only be restored from proven replicas; the original snapshot must not be fabricated. Training data deletion propagation requires data that should be deleted to eventually disappear. Conflicts between legal holds and deletion rights are decided by an authorized governance layer; the maintenance program only executes versioned decisions.