Preview Environment Lifecycle: Isolated Seeds, Lease Fencing, and Destruction Proof

Automatically deploying a URL for every PR update looks like a developer experience upgrade, but underneath it involves three sets of identities: the code commit, the workflow attempt, and the actual resources. An old attempt may resume writing DNS after network connectivity is restored, and a new environment may be deleted by an old cleanup job; two Preview environments may share a “test database,” making demo actions visible to each other; after a PR is closed, its domain disappears while cloud disks and keys continue incurring charges. CI/CD Pipeline Design describes the release stages; this article treats the Preview itself as an ephemeral object that must converge.

A Kubernetes Namespace provides resource-name isolation, but it does not automatically isolate cluster-scoped objects, external databases, or shared buckets. The Lease API provides a holder and renewal time, but the business layer must still decide who may write. GitHub Deployment connects a ref, an environment, and a status, but it does not mean that resources have been created or destroyed completely. Therefore, a Namespace name, deployment status, or URL cannot individually serve as the source of truth for an environment.

One Identity Controls Every Side Effect

When the controller first accepts a PR, it creates an immutable preview.environment.v1, assigns a nonreusable environmentId, and fixes the repository, PR identity, owner, creation time, and hard TTL. The revision, artifact, seed, state, and lease epoch can change and must reside in a separate preview.control.v1, using recordVersion for CAS. Advancing the generation is a control-record event; it must not re-sign or rewrite the identity manifest.

schema: preview.environment.v1
environmentId: pe_01JZ...
expiresAt: 2026-07-20T10:00:00Z
---
schema: preview.control.v1
recordVersion: 18
generation: 4
revision: 91a6c2e...
artifact: registry.example/web@sha256:82c...
seedSnapshot: sha256:d4b...

Control-record states are limited to REQUESTED -> PROVISIONING -> SEEDING -> READY -> DRAINING -> DELETING -> DELETED; when a side effect cannot be confirmed, the record enters QUARANTINED. The reconciler writes only to a transactional outbox, and only the single effect executor may call providers. Each entry stores its effectId, generation, payload digest, status, and returned providerOperationId. The executor prioritizes provider-native idempotency keys and ETag/precondition mechanisms. When these are unsupported, it creates a unique resource name for each generation; reusing a name and overwriting the resource is forbidden. An entry that times out and whose effect cannot be confirmed enters UNKNOWN_EFFECT; it must not retry the opposite operation or advance to READY.

A shared Preview DNS alias is serialized through a separate alias CAS/Lease. Only the current control generation may point it to a generation that has reached READY, and the actual hostname of every generation is never reused. A PR update advances the generation with CAS and inserts the outbox entry in the same database transaction. An old executor may only append its own provider receipt; it cannot update the current state or alias. After closure, revoke entry points and credentials first, stop new sessions, and then delete resources according to the ownership inventory.

A Seed Is Not a Mutable Script

Seed data is published as an immutable seedBundle.v1 containing the schema version, generator revision, random seed, fixture blob digest, row-count invariants, and sensitive-data attestation. A Preview may load only an approved digest and must not sample production data at runtime. Email addresses, phone numbers, and payment identifiers are generated within explicitly defined synthetic domains. When relationships across tables are required, use stable IDs within the bundle rather than ordinary hashes of production values. Databases, caches, search indexes, and object storage are all partitioned by environmentId, and the audience of application credentials is restricted to that identity.

Loading uses a staging area: first verify the digest and schema, complete every row-count and referential-integrity check, and then atomically switch the data to readable. A partial bundle cannot enter READY. Database Migration Management handles backward and forward structural compatibility; the seed gate separately verifies whether the current application version can read this semantic data. Refreshing a seed creates a new generation through control CAS instead of clearing an environment in place while it is being demonstrated. If a team needs to preserve a scenario, it should export a reviewed synthetic delta rather than extend the lifetime of a production-data copy. Short-lived credentials from Secrets Management must also be bound to the Preview audience; a general-purpose staging token must not be reused.

Replay, Concurrency, and Destruction Failures

The fault suite concurrently sends duplicate webhooks, delays an old attempt by 15 minutes, kills the executor after the provider succeeds but before the acknowledgment, immediately reopens a PR after it is closed, kills the process when the seed upload reaches 60%, stalls a finalizer while deleting a Namespace, and makes an old cleanup process touch a new generation. It must also swap the database URIs, secret audiences, object prefixes, and DNS aliases of two environments. The expected outcomes are idempotent outbox replay, UNKNOWN_EFFECT, STALE_GENERATION, PARTIAL_SEED, QUARANTINED, and ISOLATION_VIOLATION.

Every resource must carry an immutable owner tag: environmentId/generation/effectId. When an old generation enters UNKNOWN, the reconciler must use that tag to conduct a complete inventory across the provider, cluster-scoped objects, DNS, databases, and object storage, adopt the providerOperationId, and only then compensate. Failure to find a resource does not mean it was never created. A destruction receipt bound to the generation and scan digest may be issued only after all known entries have been deleted and three consecutive estate-wide owner-tag scans, separated by at least five minutes, all return zero. Unknown or untagged resources can only enter QUARANTINED.

Acceptance gates require the following: 100 rounds of concurrent creation produce only one current environment; the number of successful cross-environment reads and writes is 0; the interception rate for 10 classes of late operations is 100%; replay after a lost acknowledgment does not create a second resource; during concurrent switching, the shared alias always points to the current READY generation; seed-invariant coverage before READY is 100%; entry points and credentials become invalid within two minutes of closure, and 99% of environments complete three zero-result scans within 30 minutes; the resource count in each destruction receipt exactly matches the inventory.

Rollout begins by having the new controller generate outbox plans in shadow mode without executing side effects. It then manages only 5% of Preview environments for internal repositories before gradually taking over DNS, seeds, and deletion. During migration, the old and new controllers must not hold the executor Lease simultaneously. If the creation failure rate exceeds 2%, an isolation check fails even once, or P95 readiness time degrades by 30%, freeze new environments, transfer the Lease back to the old executor, and reconcile the new controller's generations by owner tag. Feature Toggle controls the exposure of user functionality; rollback here must also transfer resource ownership and destruction responsibility.