Publishing Training Shards: Content Digests, Set Integrity, and Atomic Cutover

Of one hundred training shards, ninety-nine may upload successfully, and every successfully uploaded file may have the correct hash, yet a prefix listing may still cause the job to start immediately. In another failure mode, after an alias is switched, workers that started early read the old part-003, while workers added during scale-out read the new file. A per-file checksum prevents neither a missing member of the set nor a single run mixing two releases.

Browser Model Asset Integrity protects Wasm, weights, and client caches, while Signed RAG Index Releases protects online query indexes. This article focuses on a trainer's input shards: beyond trusted bytes, it must also prove that the sample set, Schema, worker barrier, and checkpoint recovery are consistent.

Descriptors Bind Stored Bytes and Logical Content

Each shard uses an OCI descriptor-style mediaType + digest + size and records the immutable object-storage version. The compressed file's digest covers the bytes actually stored, while the decompressed canonical record stream receives a separate logicalDigest; otherwise, recompression could be mistaken for the same object, or a compressed stream whose bytes are valid but whose records fail to decode could still be admitted.

{
  "schema": "training.shard.release.v1",
  "datasetSnapshot": "sha256:8a91...",
  "tombstoneWatermark": 44192,
  "benchmarkFence": 3908,
  "recordSchema": "sha256:7d11...",
  "shards": [{
    "id": "shard-00017-of-00480",
    "mediaType": "application/vnd.apache.parquet",
    "digest": "sha256:31ef...",
    "size": 8388192,
    "objectVersion": "gen-9021",
    "logicalDigest": "sha256:a41c...",
    "records": 10000,
    "tokens": 8194421,
    "memberSetDigest": "sha256:51b0..."
  }],
  "totals": {"shards": 480, "records": 4800000},
  "orderedMemberTupleRoot": "sha256:b88d..."
}

Training Dataset Snapshots defines logical membership. A shard release must prove that every member appears exactly once and that its content has not changed. The verifier parses every record, forms a canonical (sampleId, sourceId, sourceVersion, logicalContentDigest, split, schemaDigest) tuple, builds a multiset, and performs a full diff against the snapshot. The root manifest's orderedMemberTupleRoot binds that set, the tombstone watermark, and the benchmark fence. Missing, extra, and duplicate members, content substitutions, and split conflicts must all be zero.

Schema, serializer, sort seed, sharding function, compression parameters, and key version are all included in the root manifest. The report from Training Data Schema Evolution is bound to the same digest. After decryption, the verifier checks the logical digest; the release is rejected if the key, AAD, or reader does not match.

Write to Staging, Then Switch Only the Pointer After Validation

The writer writes to staging under digest-addressed paths using conditional creates; overwrites are prohibited. An independent verifier recomputes the size, storage and logical digests, footer, counts, and tuple multiset from the exact object version. The report and SLSA provenance are bound to the root digest; approvers do not approve mutable directory names.

Publication, rollback, run and checkpoint admission, and advancement of the tombstone/benchmark fences all contend for the same linearizable gate. The gate compares both kinds of watermark in the current state and the root, verifies zero hits, checks the alias generation, and then issues a permit bound to root/watermark/benchmark/generation. Advancing either fence immediately closes affected permits; a CAS retry is idempotent only when it uses the same digest. Copying files cannot simulate atomicity.

When creating a run, the coordinator resolves the active pointer exactly once and sends the exact root digest to every worker. Each worker verifies its object and reports (runId, rootDigest, shardDigest); step 0 begins only after every rank agrees on the same root. Scaling and recovery change only the assignment and never reread the alias. Kubernetes Model Weight Distribution provides a useful pattern for cache validation, but training shards also require a global tuple proof.

Deliberately Induce Partial Releases and Version Mixing

Fault injection includes truncating or swapping a shard, replacing content/sourceVersion/split under the same ID, omitting a descriptor, stale listings, old caches, mixed Schemas, barrier timeouts, and recovery that mistakenly reads the new alias. A candidate must remain in STAGED/INVALID, or the entire run must fail before step 0; a replacement worker must not read another version.

Metrics are aggregated by release digest and cover the percentage of verified shards, missing/extra/duplicate members, digest failures, cache corruption, barrier wait time, worker root mismatches, read throughput, and decompression ratio. The ready probe succeeds only after the process has pinned and verified the root manifest; it does not treat object-store connectivity as proof that the data is ready. Training logs retain digests and error codes, not sample text.

The previous release is retained until the rollback window ends. A rollback points a higher generation at the old root; both the tombstone and benchmark watermarks of that old root must satisfy the current gate, or it is ineligible. An existing run must either finish with its input pinned or be canceled and recreated; shards cannot be swapped mid-run. Atomic publication requires the bytes, tuple set, and every worker to point to the same approved digest.

Further Reading

• Browser Model Asset Integrity
• Signed RAG Index Releases
• Training Dataset Snapshots
• Training Data Deletion Propagation