Sequential Testing for Pairwise Model Comparisons: Preregistration, Peeking, and Multiple-Comparison Control

A team compares A and B on the same set of questions, checking the p-value after every 50 items. If the result is significant at item 150, it declares B the winner; otherwise, it continues to 500 items. The error-rate guarantee of an ordinary fixed-sample test assumes only one look. This optional stopping increases the chance of an accidental "victory." If the team also compares five candidates and publishes only the best one, multiple comparisons amplify the problem further.

The LLM Application Evaluation Framework explains why business tasks matter more than public leaderboards, while The Coding Agent Tool Selection Benchmark freezes repository-level tasks and blind review. This article focuses specifically on how pairwise outcomes for the same item can produce an acceptable conclusion under a sequential plan. It does not discuss whether the Judge itself is qualified.

Register the Comparison Plan Before Seeing Any Results

Publish an immutable pairwiseComparisonPlan.v1 that fixes the baseline/candidate release, item membership root, pairing unit, replicate seed, A/B presentation-order randomization, Judge or human rubric, primary outcome, tie/abstain/UNKNOWN and paired-censor rules, provider request lookup, target estimand, minimum practical effect, alpha/power, maximum sample size, looks, sequential method and boundaries, futility rules, candidate family, and multiplicity adjustment. Changing the tie algorithm or removing difficult items after seeing interim results must create a new experiment.

schema: pairwiseComparisonPlan.v1
experiment: model-b-vs-a@3
itemRoot: sha256:9ac...
unit: item
outcome: [A_WIN, B_WIN, TIE, INVALID, OUTCOME_UNKNOWN]
looks: [200, 400, 800]
alpha: 0.05
spending: precomputed-boundary@sha256:30e...
maxN: 800
family: q3-candidate-set@4

Wald's original paper established the fundamental ideas of sequential testing and its error boundaries. NIST's multiple-comparisons guidance explains why inference across multiple groups cannot reuse the nominal error rate of a single comparison. In practice, you can choose group-sequential alpha spending, always-valid e-values, or another validated method, but the calculator version, assumptions, and applicable distribution must be recorded in the plan. Recomputing a fixed-sample p-value every hour cannot be repackaged as a sequential test.

Generate A and B together for the same item, binding them to identical input, tool fixtures, and applicable random conditions to form a pair slot. Randomize the A/B presentation order according to the preregistered rules, then map the Judge receipt back to the actual release. LLM Judge Calibration constrains position and style bias. Bind each side's call to a stable attempt/provider request ID. If a response is unknown, first query it under the same ID; do not replace the ID or rerun only one side. The entire pair may be recorded as INVALID/censored only if the plan preregisters paired censoring and evidence proves that the attempt never occurred or is unrecoverable. Otherwise, retain OUTCOME_UNKNOWN and block the look. Confirmed failures may also be retried only as a pair under the plan. For user-level repeated tasks, estimate uncertainty by user or session cluster; correlated requests cannot be treated as independent samples to inflate n.

Each Look Has Exactly One Acceptance Point

The experiment state is REGISTERED -> COLLECTING -> LOOK_READY -> CONTINUE | STOP_EFFICACY | STOP_FUTILITY | MAX_SAMPLE -> SEALED. Each side's attempt follows REQUESTED -> RESPONSE_RECORDED | OUTCOME_UNKNOWN | FAILED_CONFIRMED; after UNKNOWN enters RECONCILING, it can reach a known terminal state only through the same request ID. The collector deduplicates on (experiment, item, replicate) and stores both attempt receipts, artifact digests, presentation order, verdict release, and terminal state. An immutable data root may be produced only when the cumulative number of valid pairs reaches a planned look, all earlier slots are terminal, OUTCOME_UNKNOWN = 0, and every censored slot has a paired-censor receipt permitted by the plan.

The sole acceptance point is the statistics registry's commitSequentialLook CAS. It matches the plan digest, look ordinal, cumulative pair root, attempt/reconciliation root, randomization root, Judge release, paired-censor receipt root, and candidate family generation. It requires OUTCOME_UNKNOWN = 0 in the included window, recomputes the boundary, and then writes the look receipt. If a stopping boundary is crossed, the same transaction writes the terminal decision, preventing subsequent samples from changing the winner. Reruns by two analysts or a late verdict may only trigger an audit discrepancy; they cannot select a more favorable root.

When comparing three or more candidates, close the family at initial registration and incorporate sequential and multiplicity control jointly into the boundaries. Adding C after seeing a difference between A and B, or moving a failed candidate under a new experiment identifier, still belongs to the same decision family. Exploratory results may be displayed, but they must be labeled EXPLORATORY and cannot use formal alpha language. Agent Online Evaluation Metrics handles real-world guardrails after a candidate wins; offline significance must not be equated with business value.

Manufacturing a False Winner with Optional Stopping

The fault suite peeks every 10 items in null-effect simulations, retains only significant seeds, always presents the candidate first, duplicates high-win-rate items, discards ties, drops the ACK after model acceptance, makes provider lookup unavailable, reruns one side under a new ID, changes the Judge version midway, delays negative verdicts, creates a sample-ratio mismatch, registers only the winner from a multi-candidate comparison, and makes two look writers race. The expected number of formal decisions from unplanned looks, uncleared UNKNOWN states, or censoring without evidence is 0. Post-terminal reversals and duplicate pairs entering the denominator must also remain at 0.

The gate requires the upper confidence bound on family-wise type-I error across at least 100,000 null simulations to remain at or below the preregistered alpha, with power meeting its target under the target effect. At commit time, UNKNOWN must be 0 and paired-censor receipt coverage must be 100%; deviation from a 50/50 presentation order, the pair-missing rate, and Judge abstention must all remain below their thresholds. Every decision must be traceable to the plan, data root, and boundary implementation. Agent Evaluation Dataset Design continues by checking slice coverage. A new test engine first shadow-recomputes historical experiments, then takes over one low-risk candidate family. If calibration simulations, data conservation, or online guardrails fail, the serving alias reverts to the baseline. Experiment receipts and negative results are retained, and sampling must never be reopened merely to obtain the desired answer.