Training Dataset Snapshots: Immutable Manifests, Lineage, and Reproducible Data Retrieval

A training job configured with dataset=customer-support-latest cannot tell you exactly which records the model read. Object uploads may race with directory scans, CDC may still be catching up, and results may change after cleaning code is redeployed. When a failed training run is retried, its input set may already differ even if the SQL and random seed are identical. The general concept of data versioning only reminds teams to "record a version." The problem addressed here is training admission: freezing membership, sources, transformations, and revocation watermarks into a machine-verifiable contract.

Database CDC continuously captures changes, while RAG Signed Index Release Manifests cover online-query artifacts. This article neither builds retrieval indexes nor describes browser caches. It defines only the source-data snapshots that training jobs may consume.

A Manifest Must Enumerate Facts, Not Store Query Notes

Every snapshot uses a snapshotId that is never reused. First, compute a digest over a canonical payload that explicitly excludes the snapshotId, signatures, and transport metadata. Then place the digest in the external descriptor and display fields. The verifier must recompute it using the same exclusion rules to avoid manifest self-reference. At a minimum, the manifest pins parent snapshots, source versions, watermarks, schema, policies, transformations, members, splits, tombstone and benchmark watermarks, and purpose:

{
  "schema": "training.dataset.snapshot.v1",
  "snapshotId": "sha256:8a91...",
  "parents": ["sha256:31c0..."],
  "sources": [{
    "id": "tickets-eu",
    "tableSnapshot": "iceberg:918337",
    "cdcHighWatermark": "lsn:0/8F34A210"
  }],
  "tombstoneWatermark": 44192,
  "benchmarkFence": 3908,
  "recordSchema": "sha256:7d11...",
  "selectionPolicy": "sha256:2b70...",
  "transformImage": "registry/normalize@sha256:6e4f...",
  "members": {"uri": "cas://sha256/91ad...", "count": 482109},
  "splitPolicy": "sha256:902c...",
  "purpose": "support-model-training-v3"
}

SQL text, directory prefixes, and Git branches are only hints, not proof of membership. Object sources must record immutable versions or generations and byte digests. Table formats must use a pinnable snapshot ID instead of rerunning WHERE updated_at < t later. The membership table stores stable sample IDs, source versions, logical-content digests, and splits; it does not copy sensitive source text into the manifest. The manifest is serialized using frozen field-order, Unicode, and numeric rules. Different implementations must not format it independently and then claim the same digest.

W3C PROV entities, activities, and agents can express which transformation, executed under which identity, generated a snapshot from which sources. SLSA provenance adds build identity and environment information. Lineage edges point to immutable digests; derivedFrom=latest has no audit value. Signatures from AI Supply Chain Security can prove only that a statement came from a trusted builder. The set validation below is still required to prove that the statement is true.

Freezing Starts by Establishing a Consistent Boundary

The control plane first reads the minimum watermark for deletions and authorization revocations, then obtains a transactional snapshot or supported high watermark from each source before running selection and transformation. New events arriving during the scan belong to the next snapshot. Events that predate the watermark but arrive late send the build to INCOMPLETE; they must not be silently added to an already signed set. If a source cannot provide a consistent snapshot, rebuild it from a controlled incremental log to a specified offset and prove that the offsets are contiguous. One successful directory listing is not a consistency guarantee.

The builder writes the membership list and statistics to content-addressed staging. It verifies that source versions remain readable, every record ID is unique and belongs to exactly one split, tombstone and benchmark matches are absent, and unknown origins are zero, then signs the manifest. State may advance only through BUILDING -> VALIDATED -> APPROVED -> ACTIVE. Activation, transition of a tombstone to FENCED, and advancement of the benchmark fence must compete through the same linearizable (tombstoneWatermark, benchmarkFence, aliasGeneration) gate, implemented with SERIALIZABLE, row locks, or a single conditional CAS. The alias may switch only when both watermarks equal those in the manifest and the alias generation has not changed. If either fence commits first, activation fails and a new manifest is generated. Switching the alias first and cleaning asynchronously is prohibited. Training Data Schema Evolution determines whether a new reader can interpret the same members correctly.

The training API accepts only a manifest digest. Within the shared gate, the scheduler checks the purpose and the tombstone and benchmark watermarks, acquires a bound permit, and then writes the digest into the run. Workers read exact object versions and report their digests; they must not resolve latest. A resumed run continues to use its original digest. Changed data requires a new run.

Validate with Set Diffs and Replay

Fault injection includes adding or deleting objects during a scan, replacing content while reusing an object name, breaking the CDC offset sequence, assigning a different digest to the same transformation-image tag, placing the same ID in two splits, truncating the membership table, tampering with the manifest after signing, and advancing the tombstone after activation reads the watermark but before the alias CAS. In the last race, only one side may commit. If revocation wins, the candidate remains in VALIDATED, while the already pinned run input remains unchanged.

The gate records source-watermark coverage, unversioned object counts, lineage completeness, duplicate and orphan IDs, tombstone match counts, set differences, transformation replay hashes, build latency, and unexplained row counts. For a fixed canary, replay from the pinned source versions must produce exactly the same set of member IDs. If numeric transformations permit a tolerance, the tolerance and comparator version must also appear in the report. Matching total row counts alone is insufficient.

Publication first shadow-generates the new snapshot and compares its additions, deletions, split migrations, and distribution differences with the current version. Human approval is bound to the digest. The active alias switches through CAS and increments its generation. A rollback likewise appends a higher generation that points to the old digest; it never modifies a snapshot. Atomic Training Shard Publication then packages this logical member set into physical files that can be read in parallel. A rollback must never cross back before the latest tombstone watermark, or reproducibility becomes the resurrection of revoked data.

Further Reading

• Database CDC
• AI Supply Chain Security
• Training Data Schema Evolution
• Atomic Training Shard Publication