Coding Agent Tool Selection Benchmark: Frozen Repository Tasks, Blinded Evaluation, and Procurement Gates

Public leaderboards can show how a model performs on a certain frozen dataset, but they can't answer whether a candidate tool can understand your company's build system, permission boundaries, and review requirements. Live demos are even easier to skew: the vendor picks familiar tasks, engineers give one tool extra hints, failed runs get restarted, and in the end only the best attempt is shown. AI coding tools comparison gives an overview of product shapes; this article only addresses how to generate auditable, recomputable repository-level comparison evidence before procurement.

Freeze benchmarkPortfolio.v1

The decision owner first publishes the immutable coding.toolBenchmarkPortfolio.v1. Each taskId binds an authorized repository, base commit, a problem-statement summary, allowed paths, the build image digest, the dependency cache strategy, network and tool permissions, time and token budgets, a hidden acceptance oracle, a risk level, and a contamination check. Tasks cover defect fixes, small features, cross-package refactors, test backfilling, and read-only diagnostics, but you must not hand real undisclosed vulnerabilities directly to a data boundary that isn't allowed.

schema: coding.toolBenchmarkPortfolio.v1
portfolio: platform-q3@4
taskId: checkout-timeout-017
baseCommit: 91af0c2
runnerImage: sha256:7b1...
budget: { wallSeconds: 1800, modelUsd: 8 }
oracleBundle: sha256:34d...
accessProfile: repo-write-no-network.v2

SWE-bench shows the basic shape of using real repository issues and tests to evaluate fix capability, but an enterprise benchmark must also handle local dependencies, private interfaces, and permissions. Task authors and tool operators are separated; the oracle is sealed before execution and is not readable by candidates. If a task has already entered model training, has a public fix, or is in a vendor's tuning set, it is flagged CONTAMINATED and removed from the formal score; you can't keep using it by changing the issue title.

The portfolio must be stratified by repository language, size, and change type. Choosing only ten easy-to-fix bugs overestimates automation value; choosing only historical incidents underestimates daily throughput. Each stratum pre-specifies the sample size and minimum capability, without adjusting weights after seeing results. Any change in task revision, environment, or oracle generates a new generation, and old results cannot be mixed into the new combination.

Same acceptance boundary, not the same prompt

Candidate tools have different interaction shapes, and forcing the use of exactly the same prompt is actually unfair. The fair condition should be the same task materials, permissions, budget, starting tree, and acceptance boundary. Each run records the tool and model revision, a configuration digest, input materials, command receipts, the patch tree, elapsed time, cost, manual intervention, and termination reason. Telemetry can reference a fixed revision of the OpenTelemetry GenAI semantic conventions, but its current status is still Development, so adopters must lock the schema; high-cardinality code and prompts are saved only as controlled digests. Coding agent command execution receipts prove command facts, and change validation gate continues to judge whether the candidate patch satisfies repository checks.

For the same task, all candidates run in random order, with at least three pre-registered seeds or independent sessions. Timeout, tool crash, refusal to execute, and requiring human rescue are all results; you can't retry indefinitely until success. Infrastructure failures may be rerun only when they are shared across all candidates and proven by a receipt; a single candidate losing state counts toward reliability. Cost is reported at the same time for vendor fees, runner resources, and manual minutes; you can't compare only token unit price. AI cost attribution provides request-level material, and this benchmark merges it into the task results.

Independent oracle and blinded review

The machine oracle first proves on the base commit that it fails, then proves on a known-correct fix that it can discriminate; tests, static rules, and runtime assertions are all bound to the oracle digest. After a candidate patch passes the hidden checks, a reviewer who does not know the tool's identity checks out-of-scope changes, maintainability, and risk. The W3C PROV entity/activity relationships can connect task, run, patch, oracle, and verdict, but the final winner must come from versioned rules, not a screenshot of a chart.

Results are at minimum classified as SOLVED / PARTIAL / INVALID / TOOL_FAILURE / INFRA_FAILURE. The main metric uses the stratified task-resolution rate and confidence interval; it also reports the time-to-first-valid-patch, manual intervention, out-of-bounds writes, verification pass rate, and unit cost of success. Non-deterministic tools should use paired bootstrap or a pre-registered stratified model, and not treat three runs as three independent repository tasks. Agent replay evaluation checks control-flow degradation; this article compares the delivery results of complete tool products under a fixed work contract.

Attack the selection conclusion with bias injection

The fault drill includes: pre-warming the index for one candidate, leaking hidden tests, recording cache-hit costs as zero, rerunning only the failed candidate, swapping model revisions, loosening network permissions for a certain tool, letting the reviewer see the brand, deleting timeout runs, and updating the extension mid-evaluation. The system must separately report UNEQUAL_ACCESS, ORACLE_EXPOSED, COST_INCOMPLETE, SELECTIVE_RETRY, REVISION_DRIFT, or BLINDING_BROKEN, and invalidate the affected comparison.

The procurement gate requires 100% task-material coverage, 100% candidate access-profile consistency, 0 hidden-oracle leakage, and 0 run loss, plus a deterministic blinding audit confirming that brand strings, file paths, authors, patch metadata, telemetry fields, and UI styles have all been removed. Identity guessing significantly above random can only prove the blinding was broken; even if it is not significant under the pre-registered sample size, it also cannot prove the blinding was not broken. The lower bound of the resolution rate for the key repository stratum must also reach the pre-registered threshold, with 0 out-of-bounds writes. Before going live, first shadow the winning tool in read-only diagnostics and low-risk fixes, then open writes for 5% of qualified tasks. If the real resolution rate falls 10 percentage points below the benchmark, manual minutes double, or a single unauthorized write occurs, pause automatic allocation and switch back to the old tool. The benchmark, negative results, and old reader are retained until the renewal cycle; the next procurement re-freezes the portfolio rather than letting one win be permanently valid.