Training Data Schema Evolution: Compatibility Matrices, Semantic Migration, and Release Gates

Changing label from an integer to a string may still allow a file to be read successfully, while training silently learns a different class system; changing latency from milliseconds to seconds leaves the type unchanged but makes the numeric distribution 1,000 times smaller. Training-data compatibility therefore has two layers: whether the encoding can be parsed and whether the training semantics remain the same. A single deserialization test cannot protect the latter.

RAG Permission Metadata Migration focuses on preventing old indexes from bypassing ACLs at query time, while this article focuses on how offline samples are interpreted by the tokenizer, collator, and trainer. TypeScript Runtime Trust Boundaries explains that type declarations cannot validate external input; training pipelines must likewise enforce real Schema and semantic gates at the ingestion boundary.

Version Field Identity and Semantics Together

Identifying fields only by name turns a rename into "delete and add." The contract assigns a stable ID to each field and records its physical type, logical type, nullability, default value, unit, normalization, sensitivity level, allowed range, and owner. For labels, it must also pin the vocabulary digest, behavior for unknown categories, and the ignore value used by the loss:

schema: training.example.v4
fields:
  - id: 17
    name: response_latency_ms
    physical: int64
    logical: duration
    unit: millisecond
    nullable: false
    range: [0, 600000]
  - id: 23
    name: outcome
    physical: string
    vocabulary: sha256:94c1...
    unknownPolicy: reject
normalizer: sha256:12ee...

Avro's writer/reader Schema resolution rules can handle field additions and defaults, while Iceberg uses stable field IDs to support renaming and reordering. These mechanisms solve structural matching but do not automatically prove that unit, label, or missing-value semantics are correct. Adding a nullable field is compatible only if its default does not conflate "unknown" with a genuine zero. Integer widening is generally safe, while floating-point narrowing, unit changes, label merges, tokenization normalization, and array-order changes must all be treated as changes requiring explicit migration.

Each proposal generates a change record listing from/to, compatibility class, transformation-function digest, affected consumers, backfill strategy, and rollback deadline. Before deleting a field, first stop new consumers from depending on it, then wait until all supported reader versions have passed the retirement watermark; do not physically delete it merely because producers no longer write it. Tokenizer Compatibility Migration is another part of the training ABI. The tokenizer digest must enter the snapshot alongside the data Schema, but it must not be treated as merely an ordinary text field.

The Compatibility Matrix Covers Actual Read/Write Combinations

The registry stores the versions each producer can write, the range each consumer can read, and their intended uses. CI runs at minimum old writer -> new reader, new writer -> current reader, a round trip through the new version, and trainer images that remain within the rollback window. Unknown fields, unknown enums, nulls, extreme values, invalid UTF-8, oversized arrays, and missing defaults must all produce explicit errors; the parser must not automatically coerce them to zero or an empty string.

After structural checks pass, run a semantic golden set. Fixed samples pass through both the old and new transformations, with comparisons of sample IDs, splits, labels, numeric errors, token lengths, and rejection reasons. Thresholds for permitted changes must be specified by the change record. Shadow statistics compare null rates, cardinality, quantiles, class frequencies, sequence lengths, and label rates for every critical slice. Similar global means must not mask every sample in a small language or high-risk category being mapped to unknown.

A Training Dataset Snapshot pins exactly one recordSchema digest. Mixed-version source data must first be converted into a new snapshot by a versioned converter. The trainer must not guess the Schema from the first row or dynamically choose different defaults within the same shard. The backfill worker records the source snapshot, input/output counts, reject set, checkpoint, and monotonically increasing watermark. Retrying the same partition must produce the same output digest, and different conversion versions must never overwrite the same path.

Use Dual-Read Evidence for Migration Instead of Direct Replacement

The usual release order is to deploy consumers capable of reading both old and new versions, then enable dual writes or shadow conversion for producers, and finally switch to the new snapshot and retire the old writer. If dual writes span two independent transactions, they must be reconciled using stable sample IDs without claiming atomicity. A more robust approach retains one source of truth and derives the two outputs through deterministic converters. The gate requires zero unsupported readers, zero unexplained rejects, and joint approval of the difference report by the data owner and model owner.

Fault injection includes a producer writing a new enum too early, a reused field ID, unit metadata not being updated, a late old worker overwriting a backfill, half the partitions remaining on the old version, a changed default, replacement of a vocabulary file under the same name, and a rollback trainer that can read only the old Schema. Metrics are recorded by schema digest for parsing failures, unknown fields, conversion rejects, dual-read disagreements, distribution drift, remaining old readers, and the backfill watermark. High-cardinality fields must not be used as monitoring labels.

After the switch, retain the old snapshot and compatible readers until the observation window ends. When anomalies occur, use CAS to switch the training alias back to the old snapshot and increment the generation rather than modifying converted files in place to revert them. Newly produced data that the old reader cannot interpret remains quarantined. Training Shard Atomic Publishing ensures that workers do not mix physical files from two Schemas. The true completion criterion for compatibility is evidence for every supported read/write combination and critical semantic slice, not a green status in Schema Registry.

Further Reading

• RAG Permission Metadata Migration
• Tokenizer Compatibility Migration
• Training Dataset Snapshots
• Training Shard Atomic Publishing