Coding Agent Flaky Test Governance: Reproduction, Attribution, and Quarantine Recovery

A commit that is sometimes green and sometimes red does not mean "just run it again." It may reveal shared state, race conditions, timezone dependencies, unwaited async tasks, resource starvation, or external service fluctuation. When a coding Agent is rewarded for restoring green as quickly as possible, it is most likely to increase retries, extend waits, or simply skip tests, hiding real defects in the main branch.

The first rule of governance is to distinguish "product regression" from "test instability." The first failure is still treated as a real failure with full evidence retained; only when the same commit with the same input shows both pass and fail across controlled repetitions can it be flagged as a flaky candidate. General debugging methods can reference AI-Assisted Debugging, but quarantine permissions and exit conditions must be controlled by the platform, not left to the Agent to decide.

Capture Replayable Context on First Failure

CI assigns a runId to each test run, recording the commit SHA, container image digest, Node and test framework versions, CPU/memory limits, concurrency, timezone, locale, random seed, shard, test order, and external dependency versions. Sensitive environment variables retain only key names and key versions, never outputting values. Logs, screenshots, traces, network errors, and resource curves are stored as time-limited artifacts.

Using Node 22.18.0, pnpm 10.14.0, and an exact lockfile as an example, the reproduction job launches an isolated container from the original commit:

pnpm install --frozen-lockfile
node --version
pnpm --version
sha256sum pnpm-lock.yaml
pnpm exec playwright test tests/checkout.spec.ts \
  --project=chromium --workers=1 --repeat-each=30 --retries=0

--retries=0 is important: the reproduction phase needs to observe the raw failure frequency. First replay with the original concurrency and order, then separately pin a single worker, random order, constrained CPU, slow network, and different timezones to locate the trigger condition. For server-side tests, freeze the database snapshot, message queue starting offsets, and fake clock. Cross-service causal chains can be correlated with Distributed Tracing, but bodies and credentials should be redacted.

The failure fingerprint cannot be just the test name. It consists of the first business stack frame, exception type, normalized error message, key assertions, and phase; noise such as timestamps and ephemeral ports is removed. When the same test has two fingerprints, treat them as two problems, avoiding folding a new product regression into an old flaky ticket.

Decide Quarantine Based on Evidence

The candidate test is executed at least 20 times on the same SHA, input, and environment, observing at least one pass and one same-fingerprint failure. Three non-consecutive failures with the same fingerprint in the most recent 50 main-branch runs can only place it in the candidate queue; before quarantine, a failing SHA must still be selected and a pass and a failure replayed in a frozen environment. Tests involving authentication, billing, data deletion, and migration cannot be passed through by quarantine alone even if they meet the conditions; they must provide a stable replacement gate that covers the same risk.

The quarantine record contains the owner, cause hypothesis, first and most recent failure, failure fingerprint, reproduction command, linked tickets, replacement coverage, and expiration time, up to a maximum of 7 days. The test still runs in a separate queue on every commit; it just no longer repeatedly blocks the same known fingerprint. If a new fingerprint appears, it is immediately blocked as a new regression. Changes to skip, only, and retry configuration in the repository are intercepted by policy checks and cannot be merged without a quarantine record.

Playwright distinguishes first-pass, pass-after-retry, and persistent failure; these states should be fully reported. A pass-after-retry counts as a flaky event, not a green success; automatic retry is allowed at most once to collect classification evidence and must not mask the release gate. Overall patch evidence still follows Coding Agent Change Validation Gates.

Fix the Root Cause, Not Increase the Wait Time

Common root causes fall into four groups. Shared-state issues are fixed with per-case isolated data, unique namespaces, and cleanup validation; race issues use event or condition waits to replace fixed sleeps; time-randomness issues inject clocks and seeds; external-dependency issues use contract-stable local stand-ins while keeping a separate real integration gate. Resource starvation should be addressed by calibrating concurrency and container quotas, not by endlessly extending test timeouts.

When the Agent submits a fix, it must include a minimal failing sample and a before/after comparison. For example, the original SHA fails 7/100 under the trigger configuration, the patch passes 500/500 consecutively, and the full domain suite passes under the default configuration. Simply changing waitForTimeout(500) to 5000 does not count as a fix, because it does not establish the business condition being waited for.

If local reproduction is impossible, retain the production artifact and expand sampling in a fixed environment, with at most two rounds of automated hypothesis verification. The Agent must not change multiple independent synchronization points at once to get lucky; each round records the hypothesis, single intervention, and result. When the budget is exhausted, escalate to a human; the test remains quarantined or blocking, and no evidence-free "probably fixed" conclusion is generated.

The Recovery Gate Must Be Stricter Than the Quarantine Gate

Before lifting quarantine, the fix commit must show 500 consecutive passes with no same-fingerprint failure under historical trigger conditions, 200 consecutive passes under the default configuration, the relevant domain suite passing, and at least 3 main-branch runs observed. Counts, seeds, and artifacts must be bound to the fix SHA. If the deadline is still not met, escalate the owner and release risk; do not auto-renew.

Recovery must also undo the serial execution, extra logging, and resource relaxations temporarily added for diagnosis, then rerun under the production CI's concurrency, sharding, and test order. Otherwise, the team has only proven the diagnostic environment stable; the original trigger conditions may still exist.

The team tracks the number of flaky tests, failure rate, average quarantine duration, expired items, retry minutes consumed, and number of new fingerprints, and reviews them by root cause. Agent Replay Evaluation can also be used to save the triggering event sequence. The goal is not to make CI look perpetually green, but to turn every uncertain failure into reproducible evidence, hold the behavior line while temporarily quarantined, and ensure the quarantine ultimately exits.