Configuration Rollout Generations and ACK Quorums: Proving Instances Applied the Same Configuration

Operations changes a timeout from 30 seconds to 3 seconds, the configuration center reports success, and the console also shows “Published.” Half the instances update immediately through watch, while the other half remain on the old value after a disconnected watch encounters compaction. Recently restarted instances then submit ACKs using old Pod identities. Traffic jumps unpredictably between two different behavioral worlds, and simply counting “80 ACKs received” cannot show which 80 instances responded, which digest they applied, or whether they are actually healthy.

Ansible Configuration Management automates host configuration, while Feature Toggles control feature exposure. This article focuses on the acceptance protocol for live backend configuration: a stored configuration object is only a candidate. The active pointer may advance to the next generation only after target members prove, using their current boot identities, that they loaded the configuration and remained healthy.

Publish Immutable Artifacts, Not Overwritten Key-Value Pairs

configRelease.v1 binds the config ID, generation, canonical content digest, schema or validator digest, dependency versions, target selector, frozen membership snapshot, minimum application version, stage plan, ACK policy, health window, deadline, and predecessor. Sensitive values are represented only by secret version references and never enter an ACK. Content within the same generation is immutable; even a copy edit creates the next generation.

schema: configRelease.v1
config: checkout-routing
generation: 84
contentDigest: sha256:62e...
validator: routing-policy@7
membership: checkout-prod@epoch-551
ackPolicy: zone-quorum-v3
predecessor: 83
state: CANDIDATE

Instance state is restricted to DISCOVERED -> DOWNLOADED -> VALIDATED -> STAGED -> APPLIED -> HEALTHY_ACKED. Failures enter REJECTED, and instances superseded by an update become STALE. An ACK must include the instance workload identity, boot or session ID, membership epoch, generation, content digest, application artifact, validator version, monotonic apply sequence, and health-window result. Merely downloading the file does not permit an ACK, and an ACK issued before a process exits cannot represent a replacement instance with the same name.

A Kubernetes ConfigMap can be consumed by a Pod as a file or environment variable, but environment variables do not update automatically when the source object changes, and mounted updates still depend on the synchronization path. Therefore, “the API object was updated” is not proof that the application applied it. An etcd watch revision can help detect ordering and compaction gaps. A client that encounters a compacted revision must reread the snapshot instead of treating the next event as continuous history. The rollout status of a Kubernetes Deployment provides a workload rollout signal, but it cannot replace process-level confirmation of a specific config digest.

A Quorum Is Rollout Evidence, Not a Consensus Algorithm

The control plane calculates the ACK set against the frozen membership and requires each fault domain, critical role, and minimum traffic weight to meet its own threshold instead of merely counting Pod names. Newly scaled instances must not dilute the denominator currently under evaluation. Retiring a member requires a membership generation change and a recorded reason. A quorum proves only that a predefined sample is healthy; it does not prove that instances without ACKs are safe. After promotion, the router must therefore still refuse to send requests requiring generation 84 to instances running generation 83.

The sole acceptance point, promoteConfig, applies CAS to the active pointer. Within the transaction, it rechecks the candidate digest, predecessor, membership epoch, ACK policy, boot identity on every ACK, health windows, and revocation markers before moving the active generation from 83 to 84 and writing a promotion receipt. If two candidates race, only one succeeds. If the response is lost, the rollout enters PROMOTION_UNKNOWN; the publisher must read the pointer and receipt to determine the outcome instead of blindly attempting another switch.

Instances activate configuration only from the active pointer or an explicit canary assignment; they must not select “the largest generation in storage.” Stale watch events, late ACKs, and cache backfills must carry their generation. If the current pointer has already changed, they are retained only for audit. When configuration affects an external protocol, a compatibility matrix is also required. Stable bucketing from Model-Serving Canary Rollouts can be reused, while No-Code Workflow Generations shows why a single execution should remain pinned to one configuration generation.

Inject Network Failures Where Mixed Generations Are Most Likely

Fault injection covers missed watch events and compaction, partially downloaded content, a correct digest paired with a different validator, an instance restarting after ACK, two zones losing connectivity, membership changes racing with promotion, late ACKs from an old generation, a sudden error-rate increase after the health window, a lost ACK for the pointer CAS, and out-of-order old and new watchers during rollback. Assertions require zero promotions with the wrong digest, zero duplicate identities counted toward quorum, zero switches when fault-domain thresholds are unmet, and every request traceable to the instance generation that handled it.

Release gates require 100% observability coverage for target instances' config digests, 100% satisfaction of HEALTHY_ACKED targets in required fault domains, zero incompatible requests across mixed generations, watch gaps resynchronized within 60 seconds, PROMOTION_UNKNOWN resolved within 30 seconds, and canary error rates and p99 latency within preregistered budgets. Rollout begins with shadow download and validation, then expands to 1% of internal instances before proceeding zone by zone. Rollback uses a new promotion receipt to point the active pointer to the retained predecessor and increments the rollout epoch. Instance ACKs are recalculated; generation 84 must not be modified in place to contain generation 83's content, and evidence from the failed generation must not be deleted.