Mutation Testing for AI-Generated Patches: Proving Tests Kill Bugs, Not Just Go Green

AI can generate both implementation and tests in seconds, and that introduces a hidden problem: tests may simply restate the implementation. They execute the new branches, hit 100% line coverage, yet still pass after changing > to >=, removing an auth condition, or flipping a boolean. Mutation testing actively creates these small errors and then checks whether the tests can "kill" them, so it answers a more important question: does this set of assertions actually constrain the behavior the patch permits?

It is not a replacement for ordinary coverage, nor should it be run blindly across the whole repository. The coding agent's gate must bind the mutation scope to the current diff, risk declarations, and the approved test plan. The general static-check and regression layer remains the responsibility of the coding agent's change validation gate; the mutation layer only provides counterfactual evidence for critical logic.

Freeze inputs before comparing baselines

A reproducible run at minimum fixes the commit SHA, Node and package-manager versions, lock-file hash, test command, mutator version, and config hash. Taking a repo with Node 22.18.0 and pnpm 10.14.0 as an example, CI first runs from a clean checkout:

node --version
pnpm --version
sha256sum pnpm-lock.yaml stryker.conf.json
pnpm install --frozen-lockfile
pnpm exec stryker --version
pnpm exec stryker run stryker.conf.json

Mutator dependencies should be pinned to exact versions in package.json and recorded in the lock file; CI must not temporarily download latest. The report stores baseSha, headSha, the selected source files, the test set, timeouts, concurrency, and the location of each mutant. Only when these inputs are consistent are the baseline score and the patch score comparable.

The scope is produced by git diff --unified=0 BASE...HEAD, but you must not mutate only the added lines. If the agent modified one condition in a price-calculation function, the function itself plus the validators it directly affects must at minimum be covered; if a public type changed, expand to the related implementations. Pure formatting, generated files, and type declarations that can't be directly executed can be excluded, but exclusions must be recorded in the report. Repository structure and dependency boundaries can first be established with the coding agent repository map.

Don't look at a single mutation score

Mutation results are at minimum split into killed, survived, no coverage, timeout, and compile error. The gate uses layered rules rather than a single "total score above 80%":

For example, if if (actor.id !== ownerId) deny() is deleted and the tests still go green, this isn't a "slightly lower score"—it's an immediate security-gate failure. Conversely, a surviving equivalent-string mutation in log text can be flagged by a reviewer as equivalent or low-value, but it must be accompanied by the source location, rationale, and an approver; the agent cannot add it to an ignore list on its own.

Surviving mutants drive test fixes

The fix order is not to add one fragile assertion per mutant. First classify survivors as missing tests, equivalent mutations, unobservable behavior, or test-isolation problems. Missing tests should add business-observable outcomes—state, return values, persistence, and side effects—rather than asserting how many times an internal function was called. When you need to expand the input space, you can combine this with AI test generation, but generated cases still need to be explained by requirement invariants.

After each fix, re-run the same frozen mutant set; if the source changes again, the old report is invalidated and the set is regenerated. Allow at most two rounds of automated agent fixing; if high-risk survivors remain, escalate to manual review. Removing assertions, expanding mutate.exclude, increasing the global timeout, or converting failing tests to skipped is forbidden unless the validation plan explicitly records the deviation.

Control cost and false positives

At PR stage, only run affected modules; nightly jobs rotate over the whole repo; before merge, force a full mutation run on security and finance directories. The cache can only reuse compiled artifacts from the baseline commit; mutant results from another diff cannot be treated as current evidence. If run time exceeds 12 minutes, the gate should report "budget exhausted" and schedule a controlled full CI run instead of counting un-executed items as passed.

First calibrate the configuration on known-fault samples: prepare four classes of patches—out-of-bounds, null, permission reversal, and bad rollback—and confirm the mutator generates relevant mutants and the tests fail. If critical faults cannot be expressed, add manual fault injection or targeted tests rather than advertising the mutation score. Evidence retention and re-run strategy for the full pipeline can reference CI/CD pipeline design.

The final artifact should include a machine-readable report, a survivor list, equivalent-item approvals, commands and versions, elapsed time, and a rollback judgment. The value of mutation testing is not to produce a prettier percentage, but to let reviewers see that AI-written tests really do go red when faced with an implementation that is "almost correct but actually wrong."