Code Sample Drift Gates for Developer Docs: Version Matrices, Executable Proofs, and Rollback Aliases

The most dangerous state for a code sample is not an obvious error, but code whose syntax still compiles while its semantics are already obsolete: an SDK method retains an old overload, but the sample passes the amount in the wrong unit; the documentation site hides initialization code, leaving imports missing when users copy the visible portion; and after release, latest points to another major version. AI-Powered Product Documentation Sites focuses on content generation and retrieval; this article answers only one verifiable question: can every executable code block currently displayed on the page still complete its stated task with the version combination it declares?

This also differs from Evidence Ledgers for SEO Content Refreshes: update timestamps and query drift cannot prove that a sample runs. It also differs from Drift Gates for API Mocks and Fixtures, which governs whether test doubles represent the producer; here, the focus is on the public snippets users actually copy to their local environments and whether the rendering pipeline preserves them faithfully.

Give Every Visible Sample a Stable Identity

The documentation build first emits an immutable docs.sampleSet.v1. Each sampleId binds the source file and marker, normalized source digest, digest of the rendered copyable text, documentation revision, SDK artifact digest, dependency lock digest, runtime image digest, API revision, required scope, environment variable schema, scenario ID, and the version and digest of an independent oracleBundle. A sample can retain its identity when moved; any change to the code, hidden prerequisites, or oracle reference must create a new generation, and overwriting old proof is prohibited.

schema: docs.sampleSet.v1
sampleId: node.widgets.create.v3
sourceDigest: sha256:91c...
renderedCopyDigest: sha256:113...
sdkArtifact: npm:@acme/[email protected]#sha512-...
runtimeImage: sha256:83a...
apiRevision: widgets.v3
scenario: create-and-read-back
oracleBundle: api-conformance.widgets@7#sha256:5ce...

SemVer can express versioning intent for a public API, but it cannot prove that a particular method still behaves the same with real credentials, in a given region, or under default configuration. Version ranges are therefore used only to generate the test matrix and cannot replace execution. ^4.1 must be resolved to the exact version used for testing and recorded in the proof; historical documentation must pin the original artifact rather than drift with the registry's latest. REST API Versioning provides an API compatibility strategy; the sample manifest applies that strategy to pages, SDKs, and execution environments.

Validate in Two Stages, from Markdown to Server Results

In the first stage, a structured parser re-extracts the text users can copy from the final HTML, creates a project in an empty directory with no repository context, installs locked dependencies, compiles it, and runs it. Hidden bootstrap code is allowed only when the page clearly provides a complete download entry and its digest matches; CI must not secretly use a helper that readers cannot see. Shell, PowerShell, and Windows paths are executed separately. Outputs are normalized according to declared rules to remove nondeterministic fields such as request IDs; amounts, ordering, pagination, and error codes must not be normalized away.

In the second stage, short-lived sandbox credentials are used to call the API artifact pending release, after which a separately versioned oracle bundle verifies business invariants beyond the OpenAPI structure, such as reading back a resource after creation, replaying an idempotency key without duplicate writes, and reaching the end of pagination. The real service itself is not the oracle: the bundle is maintained and approved by the API conformance owner, and the documentation generator has no authority to generate or modify it; any change that modifies both a sample and its oracle must receive independent approvals across the two ownership boundaries. An offline simulator can make PRs fail quickly, but it and the actual service cannot prove each other when generated by the same process; a daily canary must run against the real release combination. If the API or oracle is unavailable, the result is INFRA_UNAVAILABLE; the last green result cannot be reused as if it were new proof.

Each run emits docs.sampleVerification.v1, containing the sample set and oracle bundle digests, the approval generation from each side, runner version, the complete resolved dependency graph, environment class, start and end times, step-by-step results, a summary of server-side operations, and log-redaction counts. The proof uses artifact-lineage-style subject/material relationships so reviewers know exactly which rendered result and which set of independent assertions were verified, rather than a similar file next to the source. AI Supply Chain Security explains dependency risks; this article only pins approved artifacts and build materials into the sample proof.

Deliberately Create Drift That "Doesn't Look Broken"

The mutation suite should remove a visible import, upgrade the SDK by one incompatible major version, move the default endpoint, revoke a required scope, change nullable to absent, change the amount unit, make an asynchronous flow exit before flush, make a pagination cursor repeat, put Bash quoting into PowerShell, and make the CDN return stale HTML paired with a new download package. It should also make the generator update only the displayed code while omitting the downloadable project, make the server return a schema-valid but semantically incorrect business value, and attempt to relax both the sample and its assertions in the same PR. The gate must attribute failures to the source, renderer, package, runtime, API, or oracle, and must reject an oracle digest without independent approval rather than reporting every case as "sample failed."

Release acceptance criteria are: 100% extraction coverage for all visible snippets marked runnable; a 100% pass rate for the matrix covering the current SDK and the previous major version still promised support; 100% digest agreement between HTML copyable text and tested text; 100% independent approval coverage for samples and oracles; at least 98% of prescribed scenarios complete within 60 seconds in a fresh environment; a 100% mutation detection rate, including coupled relaxations; zero undeclared outbound connections; and zero secret leaks in logs or artifacts. Languages that temporarily cannot be verified must show an explicit "unverified" state and block the default Quickstart; a green result from another language must not be used as a substitute.

At launch, first run the new runner in shadow mode against existing pages for one week, producing only diff reports; then enable the new renderer with a proof generation for internal traffic before expanding to 10% of public requests. If copy digests differ, any supported matrix fails, or p95 execution time doubles, atomically switch the documentation alias back to the previous proven sample set while retaining the failed page and proof for reproduction. A fix must publish a new generation; it must not silently rewrite archived pages or delete red records. Feature Flag Rollouts can control entry points, but eligibility is always determined by the immutable proof defined here.