Test Impact Analysis for Coding Agents: Auditable Selective Regression Gates

Coding agents frequently submit small patches. Running a two-hour full regression suite for every change makes feedback too slow for iteration, but running only tests in the same directory can miss reverse dependencies, runtime plugins, configuration, and data migrations. The goal of test impact analysis is not to "run as little as possible," but to produce a justified, reproducible, and expandable test set for a given level of risk.

The selection result must be part of the evidence for merging, not a black-box optimization hidden in CI logs. For general validation principles, see Change Validation Gates for Coding Agents. This article focuses only on why a particular test set covers a diff and when that conclusion becomes invalid.

Establish Change Facts from a Frozen Baseline

The analysis inputs are the BASE_SHA and HEAD_SHA checked out by CI. Do not trust a file list reported by the agent. Preserve rename and deletion information first, then normalize the paths:

mkdir -p .artifacts
git diff --name-status --find-renames "$BASE_SHA...$HEAD_SHA" \
  > .artifacts/changed-files.txt
git diff --binary "$BASE_SHA...$HEAD_SHA" \
  > .artifacts/change.patch
node --version
pnpm --version
sha256sum pnpm-lock.yaml tsconfig.json

The example execution environment is fixed at Node 22.18.0 and pnpm 10.14.0, with dependencies installed from the lockfile using pnpm install --frozen-lockfile. The report is bound to both SHAs, the analyzer version, the build graph hash, and the test manifest hash. Any edit to HEAD after test selection immediately invalidates the result.

The static graph creates reverse edges from imports, package dependencies, generated-code sources, and test declarations. Dynamic edges come from coverage records produced by the latest full regression run and help detect dependency injection, route registration, and reflective calls. Use the union of both: the static graph may be conservative, while dynamic coverage may miss branches that were not executed. Neither is sufficient alone. Build graph ownership, entry points, and generation relationships can be incorporated into the Coding Agent Repository Map.

Jest's --findRelatedTests can serve as one signal for JavaScript repositories, but not as the final source of truth. Build systems such as Bazel can use reverse-dependency queries to identify affected targets. For every selected test, the selector reports a reason such as direct-import, reverse-dependency, dynamic-coverage, or policy-expansion, while also listing changes it could not resolve.

Define Mandatory Expansion Rules for Off-Graph Effects

Changes to the following files cannot be handled through ordinary dependency edges alone: lockfiles, compiler configuration, test runner configuration, environment schemas, authorization policies, shared database migrations, route registration, code generators, and base images. These files can alter many targets or runtime semantics, so repository policy must expand the selection to the corresponding workspace. If the scope cannot be determined, run the full regression suite immediately.

High-risk directories also require a minimum scope. Even if the graph finds only one test for changes involving authentication, billing, data deletion, or public APIs, run at least the complete domain test suite, contract tests, and negative cases. When files are deleted or renamed, query their old reverse dependencies from the BASE graph because the nodes no longer exist in the HEAD graph. If dynamically loaded strings, globs, SQL, templates, or feature flags cannot be resolved, mark them as unknown-impact and prevent the fast gate from approving the change automatically.

Balance Speed and Confidence with Two-Tier Gates

The PR fast gate requires all direct tests and first-order reverse dependencies to pass; the selector must report no unknown paths; at least one selected test must execute a modified line; static checks and the build must pass; and tests mapped to historical defects must not be excluded. If production code changes but zero tests are selected, the analysis has failed. It does not mean that no tests are needed.

The merge queue then runs an expanded set, such as all reverse dependencies, cross-package integration tests, and a fixed sample of tests that were not selected. Run a full regression suite at least once nightly or daily on the main branch, and feed the results back into the selector. If a failing test was not included in the PR set, record a false negative. Automatic merging can resume only after the graph or policy has been corrected. Pipeline layering and artifact retention can be implemented with GitHub Actions CI/CD.

Efficiency can be measured, but test savings cannot be the only metric. At minimum, track the selected-test ratio, P50/P95 feedback time, zero-test selections, unknown-impact occurrences, failures missed by selection and found by full regression, and graph age. A release gate should require zero false negatives in the last 30 days, a graph no older than the latest successful full run, and zero unknown impacts. Otherwise, automatically fall back to full-regression mode.

Protect Coverage First When Failures Occur

When a selected test fails, the agent may fix the implementation or update the assertion if the requirement genuinely changed, but it must not rerun a smaller set merely to obtain a passing result. Any modification to tests, configuration, or production code triggers a fresh impact calculation. If a test is flaky, move it into an evidence-based quarantine process while continuing to protect the same behavior through an alternative gate or a full-suite escalation. Do not simply remove it from the result.

If the selector fails, the graph database is corrupted, coverage artifacts are missing, BASE is absent from repository history, or a query times out, run the full regression suite by default. Higher cost is an acceptable degradation; automatically approving a change without evidence is not. If even the full suite cannot finish within the merge budget, block automatic merging and send the decision for human review instead of treating the timeout as a pass.

Before deployment, calibrate the system with 20 real historical patches and several fault injections: change a shared type, delete an export, adjust a feature flag, upgrade a dependency, alter a migration, and rename a dynamic plugin. Verify that the correct tests are selected. Then continuously compare the selected set against the full results from CI/CD Pipeline Design. Trustworthy selective regression is not merely a faster command. It is a control system that detects its own omissions and proactively expands scope when uncertainty appears.