Coding Agent Requirement Acceptance Traceability: Linking Acceptance Claims to Diffs and Test Evidence

A coding Agent most easily delivers a patch that "looks done": all tests green, a fluent description, but some exception branch in the original requirement is unimplemented, or unapproved behavior is added along the way. The cause is not a lack of tests, but that requirements, code, and results share no common identity. The reviewer can only guess via natural language why a test is sufficient to prove a requirement.

Spec-Driven Development addresses how to write intent clearly, and Coding Agent Change Validation Gates determines which checks to run; this article only handles the narrower question: how an acceptance claim can be traced bidirectionally to the implementing diff, a fail-capable test, and a result bound to the same repository state.

Freeze acceptance-claim.v1

Before a task enters implementation, split the requirements into atomic claims with stable claimIds. The contract contains at least specId, specVersion, claimId, claimRevision, normalized conditions, expected results, risk level, applicable interface version, allowed evidence types, and supersedes. Gherkin's Given/When/Then can express scenarios, but scenario titles or line numbers change and cannot serve as long-term identity.

schema: acceptance-claim.v1
specId: avatar-upload
specVersion: 4
claimId: AC-017
claimRevision: 2
given: authenticated user and a payload larger than 5 MiB
when: POST /avatars is invoked
then: response is 413 and no object is persisted
evidence: [integration-test, storage-probe]

Changing expected behavior must bump claimRevision, retaining the old version and approval records; if only the wording changes but the normalized semantics do not, the revision can be retained. Adding, deleting, or weakening claims after implementation begins are all contract differences and cannot be "fixed" by the Agent silently rewriting tests. The evolution of the interface schema is still judged by Agent API Contract Compatibility Gates; this contract proves the current task's commitments, not all consumer compatibility.

Build a Bidirectional Evidence Graph

The evidence graph uses W3C PROV's entity, activity, and derivation concepts, but nodes must resolve to verifiable artifacts. An evaluation records baseCommit, headTree, uncommitted patch summary, claim contract summary, test source blob, execution configuration summary, and result summary. Edges have at least four types: CLAIM_COVERED_BY_TEST, TEST_EXERCISES_DIFF, RUN_EVALUATES_TEST, and RESULT_SUPPORTS_CLAIM.

A forward query starting from AC-017@2 should find at least one test that first fails on a wrong implementation and passes on the current headTree, then the diff hunk or generated artifact responsible for the behavior. A reverse query starting from each semantic diff hunk should find the claim it serves, a necessary infrastructure justification, or an explicit NON_BEHAVIORAL classification. This discovers both "requirements with no coverage" and cases where the Agent added permissions, defaults, or telemetry beyond the requirements.

Do not treat file paths as implementation evidence. Renames invalidate paths, and one test may traverse many unrelated lines. Nodes should store Git blob IDs, hunk context summaries, and symbol identities; generated code connects to the generator and its inputs, without fabricating claims line by line. The dependency impact surface can be provided by Repository Dependency Graph Change Planning as candidate edges, but the final edges must still be confirmed by test hits, call observations, or human justification.

Evidence Strength Cannot Rest on Coverage Alone

Line coverage only shows that execution passed through a line; it cannot prove that an assertion verified the claim. Each high-risk claim needs at least one behavioral assertion and one counterexample: after deleting the corresponding branch, flipping a comparison operator, or replacing a return code, the test must fail. For rules suited to a broad input space, add AI-Generated Code Property Testing, but property tests must still declare which claims they support and cannot substitute a random sample count for a relationship.

Test results bind to headTree and the test blob; if the Agent modifies source or tests after a green result, the relevant RESULT_SUPPORTS_CLAIM edge immediately becomes stale. Skips, pass-after-retry, unavailable external dependencies, and running only a test subset are all first-class results and must not be collapsed into a green check. Human acceptance can serve as evidence, but it must record the reviewer, input, observation, and validity period, not be written as "looked at it".

Calibrate the Graph and Gates with Fault Injection

In shadow tasks, inject in turn: deleting the test mapping for a claim; binding the test to an old tree; keeping coverage but removing key assertions; adding a business branch with no claim; attaching the result of AC-017@1 to @2; and making the test pass on a wrong implementation. The system should report UNCOVERED_CLAIM, STALE_RESULT, WEAK_ORACLE, UNJUSTIFIED_DIFF, REVISION_MISMATCH, and NON_DISCRIMINATING_TEST respectively, rather than just showing a CI failure.

The merge quantification gates are: 100% forward reachability of active claims, 100% reverse attribution of semantic diffs, 100% counterexample pass rate of high-risk claims, 0 stale results, 0 unapproved contract differences; and at least 90% mutation or wrong-implementation kill rate for ordinary claims. At the same time, limit false precision: when a tool cannot establish a hunk relationship, it enters a human queue and must not automatically add an edge to fill the ratio.

Enable Enforcement Reversibly

First generate only the graph and sample 30 merged tasks, comparing reviewer actual judgments; then enable missing-edge alerts on one repository, and only block high-risk claims after two consecutive weeks of false positive rate below 5% and zero critical misses. Each decision records policyVersion; exemptions must have an owner, reason, and a maximum 7-day duration.

On rollback, switch enforcement mode back to alert, restore the previous claim parser and policy, but retain the issued graph, failed results, and exemptions; do not delete evidence to make an old patch appear qualified. If dual reads are needed while the new schema is being written, first prove the old reader can ignore unknown edges; on incompatibility, stop promotion and recompute, never treating "unparseable" as "pass". Ultimately, what the reviewer sees is not "the Agent says it's done", but how each commitment is jointly supported by this version of the code, this set of tests, and this real execution.