Schema Registry Consumer Compatibility Rollout: Reader/Writer Resolution, Lag Watermarks, and Upgrade Order

The team added currency to order events and gave it a default value. Schema Registry's BACKWARD check passed, so the producer was deployed immediately. The real-time consumer had already been upgraded, but a settlement job that had been offline for two weeks still used the old reader; another consumer mapped unknown enum values to the first item, so the data raised no error but was aggregated under the wrong currency. Compatibility checks answer only whether a particular reader/writer schema pair can be resolved; they know nothing about each consumer's business logic or backlog watermark.

Avro deserialization uses both the writer schema and the reader schema, with defined rules for field matching, default values, type promotion, unions, and enum resolution. Confluent Schema Registry distinguishes backward, forward, full, and transitive modes; non-transitive checks generally compare only the specified historical boundary. The platform must connect these rules to the actual deployment order. Event-Driven Architecture explains loose coupling, while REST API Versioning covers request-response interfaces; event schemas must also contend with long-retained data and offline readers.

Release a closed contract over the schema and consumer scope

eventSchemaRelease.v1 binds the subject, format, canonical schema digest, registry ID, compatibility mode, whether it is transitive, writer/reader canonical forms, logical type policy, default-value semantics, field classification, producer audience, consumer inventory snapshot, topic/partition watermark, retention period, and previous release. A field with the same name but different units, timezone, or null semantics is a semantic break and must not pass merely because the parser reports it as compatible.

schema: eventSchemaRelease.v1
subject: order-events-value
release: 18
format: avro-1.12
compatibility: BACKWARD_TRANSITIVE
schemaDigest: sha256:8c1...
consumerInventory: consumers@watermark-771
producerEpoch: 42

The inventory is aggregated from authentication records, consumer groups, batch schedules, and the data catalog. For each consumer, it records the owner, reader digest, supported fields/enums, last deployment, current offset, maximum dormancy, and business assertions. Unknown consumers or ownerless retained data put the release into COVERAGE_UNKNOWN. Scanning only currently online groups will miss month-end jobs, disaster-recovery readers, and replay tools. Training Data Schema Evolution focuses on training-set fields; this contract governs continuous event reads and writes.

Use compatibility direction to determine upgrade order

The states are DRAFT -> REGISTRY_CHECKED -> CONSUMER_TESTED -> WATERMARK_READY -> APPROVED -> ACTIVE -> RETIRED. BACKWARD means a new reader can read data from old writers, so consumers are usually upgraded before switching the producer; FORWARD allows old readers to read new writers, so the producer is usually switched first; only FULL permits a more flexible order. The actual order must still be explicit in the release, and reader conformance must run against stored historical writer schemas rather than trusting the mode name alone.

Each consumer uses sealed fixtures to read every supported historical writer generation, the new candidate, and cases involving unknown enums, nulls, and default values, then emits a business-projection digest. After passing, it receives a readiness receipt bound to the reader artifact, schema digest, and caught-up offset. The producer's sole acceptance point is a gateway CAS promotion of the producer epoch; prerequisites are 100% coverage by mandatory consumer receipts, isolation or retirement of dormant consumers, and an exact match between the current registry mode and the approved digest.

Messages carry the writer schema ID and producer epoch. When a consumer encounters an unknown ID, the wrong subject, or a reader below the safety floor, the message goes to dead letter; the consumer must not guess using the "latest schema." Schema caches are also invalidated by registry generation; the fence described in Cache Revalidation can prevent an old schema from being repopulated after revocation. API Mock and Fixture Drift notes that test fixtures must also derive from real contracts instead of manually maintained, polished JSON.

Expose cases where parsing succeeds but semantics are wrong

Fault tests remove a field without a default, change decimal scale, move the logical timestamp unit, add an enum symbol unknown to old readers, mismatch the subject ID, roll back registry configuration, roll back a consumer artifact after readiness, let a backlog span the producer cutover, use a compacted topic with missing historical samples, and have old and new producers write concurrently. Assertions must ensure that parser errors, differences in business projections, and unknown coverage all block acceptance; successful decoding cannot count as completion.

Gates can require: 100% agreement between registry compatibility and an independent resolution oracle; 100% owner/readiness coverage for the mandatory consumer inventory; zero differences in fixture business digests across all retained writer generations unless included in the approved list; zero silent fallback for unknown schemas; zero acceptance of old writers after the producer epoch cutover; and dead-letter volume no higher than preregistered thresholds, with an owner for every item. TRANSITIVE mode still cannot replace testing with real consumers.

During rollout, first have the gateway shadow-validate the candidate while continuing to send the old schema, then upgrade one cohort of internal consumers and compare dual reads, and finally enable it for topic producers in stable buckets. If an anomaly occurs, stop the new epoch and point the writer alias back to the previous schema; if events using the new fields have already been persisted, verify that old readers can read them or retain the new-reader path. Rolling back the producer alone is not enough to declare recovery. Historical schemas, fixtures, and readiness receipts remain immutable. Agent API Contract Gates can reuse the same consumer-first approach, but event retention periods usually make the compatibility window longer.