Coding Agent Debug Reproduction Evidence Bundle: From Failure Scene to Cold-Start Replay

When an Agent receives a stack trace and changes code directly, it often alters the only scene of the failure along with it: uncommitted config gets overwritten, dependencies get re-resolved, test data goes stale, and all that's left is "it seems to work on my machine." AI-Assisted Debugging explains how to provide effective context; a production workflow additionally needs independent evidence proving that another clean machine can observe the same failure on the old code and see the failure criterion vanish on the candidate code.

This is not about saving the Agent's complete thought trajectory. Agent Replay Evaluation focuses on tool selection and multi-step control flow, while the reproduction bundle focuses on the code, environment, inputs, and oracle of the program being debugged. It is also not responsible for proving that shell commands actually executed; here we save only immutable entrypoints and external command receipt references, avoiding mixing the debug scene with command auditing into one non-reusable giant log.

Define reproBundle.v1

The first time a failure is stably observed, before any fix is written, the collector generates coding.debug.reproBundle.v1. The manifest uses canonical serialization and content addressing, and freezes at least: the repository and baseCommit, submodule OIDs, tracked binary patches, the untracked file list, the lockfile, build scripts, the toolchain, the container image digest, environment keys allowed to affect behavior, fixtures, the clock/timezone/locale/random seed, external service stand-ins, the failure oracle, trace references, and the collection strategy.

schema: coding.debug.reproBundle.v1
baseCommit: 73a1d9c...
workspaceDelta: sha256:18c4...
runtimeImage: registry.example/debug@sha256:91aa...
fixtures: cas://sha256/44e2...
entrypointDigest: sha256:6fa0...
commandReceiptRefs: [receipt:01J...]
oracle:
  type: structured-event
  fingerprint: NullInvoice/resolveTax/v2
  mustObserve: 1
  forbiddenAfterFix: true

A Git bundle can carry the reachable refs and objects needed for reproduction, but it will not include workspace modifications; therefore the dirty diff, untracked fixtures, and file modes must be enumerated as independent blobs and digested. An OCI image is pinned by manifest, config, and layer digests; a mutable tag alone is not enough. The W3C PROV entity/activity relationship connects the scene, the collection action, the runner, and the candidate fix; an OpenTelemetry trace keeps only the relevant span IDs and necessary attributes, without copying the entire telemetry store into the bundle.

The v1 contract must spell out required fields, digest algorithms, compression format, handling of missing fixtures, and oracle interpretation. When the reader does not recognize an oracle type, it returns UNSUPPORTED_SCHEMA; it must not degrade to "non-zero exit means reproduced." A format upgrade adds a v2 manifest; old bundles remain immutable, with an offline migration report, and in-place digest edits are forbidden.

Save the Scene First, Then Minimize Inputs

The collection order decides whether the evidence is trustworthy. First freeze Git and the workspace, then copy inputs and failure artifacts, and only afterward may the Agent add logging or shrink the sample. A minimal reproduction is a derivative of the original bundle and must record which files were removed, which reducer was used, and whether the original and minimal bundles produce the same fingerprint; a hand-rewritten "similar example" cannot replace the real scene.

Secrets, personal data, and internal addresses are redacted by field type before being written to the shared CAS; stable placeholders are used only when a comparison genuinely needs them, and high-cardinality fields must not simply be kept as plaintext hashes. The manifest keeps the sanitizer version, the count of removed fields, and the digest of the redacted blob. If redaction would change the bug, the original bundle may only enter a restricted key domain, and a minimal authorization, expiry time, and destruction receipt are generated. Coding Agent Sandbox Permissions should also forbid reproduction runners from accessing the production network by default.

Cold-Start Execution with Two-Way Judgement

An independent runner starts from an empty cache, verifies every digest, checks out the exact commit, applies the workspace delta, loads the image and fixtures by digest, then invokes the frozen entrypoint. The network is off by default; when an external protocol is genuinely needed, use a versioned recorded stand-in, and treat unmatched requests as failures. The oracle prefers matching structured error codes, state transitions, database invariants, or crash fingerprints; log text can only serve as corroborating evidence. The reproduction result records the bundle digest, runner image, start/end time, oracle version, and artifact digest.

Fix verification must be an A/B of the same bundle: the old candidate satisfies the failure oracle, the new candidate satisfies the success oracle, and both run the relevant regressions required by the Coding Agent Change Validation Gate. If tests merely "pass" on a new branch but there is no old-failure evidence, the status should be UNPROVEN_FIX. If the problem is fundamentally a flaky test, it goes to Coding Agent Flaky Test Governance, but this scene is still retained as trigger evidence and must not be erased by retries.

Fault Injection and Quantitative Gates

The fault drill must proactively swap the base commit, omit an untracked fixture, tamper with one CAS blob, replace the image digest with a tag, change the locale/clock, truncate the upload, make an external stand-in return an extra field, and have the sanitizer delete a decisive error code. The system must report STATE_MISMATCH, MISSING_INPUT, DIGEST_FAILURE, UNPINNED_RUNTIME, ORACLE_UNSTABLE, or REDACTION_DAMAGE respectively, and must not uniformly show "cannot reproduce." It must also kill the collector before upload completes, confirming that no half-finished manifest ever reaches READY.

The merge gate for deterministic issues can require: 100% coverage of required fields and blob digests, 10 cold starts on each of two supported runners, the old version hitting the same fingerprint 10/10, the candidate not hitting it 10/10 with the success oracle holding 10/10, zero undeclared network requests, 100% tamper-injection detection, and a P95 rebuild time within the team-declared 15 minutes. Probabilistic races must declare the trigger rounds, seed, and minimum hit rate in the manifest, report a confidence interval, and must not be disguised as a deterministic 10/10.

Reversible Rollout and Incident Handling

At launch, first collect in shadow, comparing only manual reproduction results; then require evidence bundles for P0/P1 fixes, and gradually expand. Each stage tracks collection success rate, cold-start reproduction rate, redaction rejection count, missing-input rate, and rebuild time; only after meeting the bar for two consecutive weeks does it raise the mandatory scope. The bundle alias points to a verified digest via generation CAS; when rolling back the runner, switch only to the previous compatible reader and never rewrite old bundles.

If the collector is suspected of leaking, immediately stop new uploads, revoke decryption authorization for original bundles, and retain the event evidence of the redacted manifest; if the new runner misjudges, fall back to shadow, fix the reader, and replay the same digest. When a code fix needs reverting, use the original bundle to verify whether the revert re-triggers the old oracle. This way, reproduction evidence can support handoffs and review, and will not lose its explainability due to a tool upgrade or a single bad release.