Evaluation Label Dispute Ledger: Double-Blind Labeling, Rubric Generations, and the Adjudication Acceptance Point

“Having two people label an item once each, then bringing in a third person when they disagree” can still produce unauditable gold labels: the second labeler may have seen the first labeler’s answer, the item may have been edited midway through labeling, the adjudicator may have selected the majority without explaining the evidence, or a new rubric may have renamed old labels in place. The final evaluation score may look precise, yet it cannot show which standard the label followed, why it changed, or which disputes remain unresolved.

Agent Evaluation Dataset Design explains how to construct samples from real tasks. This article focuses only on how a frozen sample obtains a canonical label. Human Review Queues handle operational routing; label adjudication does not approve business actions, but establishes measurement facts that affect model comparisons.

Frozen Items, Rubrics, and Independent Assignment

Publish labelingRubric.v1, fixing the task definition, label type, enumerations and their order, decision criteria for each class, counterexamples, permitted abstention reasons, required evidence spans, language/domain qualifications, blinded fields, minimum number of independent labels, and adjudication rules. A separate immutable evaluationItem.v1 binds the input, candidate output, required context, source snapshot, redaction policy, and item digest. Any content revision creates a new item generation; old labels cannot migrate automatically.

schema: labelingRubric.v1
rubric: support-correctness@7
labelType: ordinal
labels: [incorrect, major_issue, minor_issue, correct]
minimumIndependentLabels: 2
evidenceSpan: required
abstainReasons: [missing_context, conflict_in_policy, language_gap]

The assignment service creates a unique assignment between each item generation and labeler, hiding the model, Prompt, experimental group, and other labeling results. The same person switching accounts, members of the same team sharing a screen, or reviewing items in a fixed A/B order all compromise independence. Each labeling receipt stores the rubric digest, item digest, labeler qualification snapshot, display order, label, evidence spans, reason code, and elapsed time, but uses a controlled pseudonym when presented downstream. AI Data Privacy Compliance constrains the purposes and retention periods for raw user content and labeler identities.

States are restricted to UNASSIGNED -> BLIND_LABELING -> AGREEMENT_CANDIDATE | DISPUTED | ABSTAINED -> ADJUDICATING -> ACCEPTED | EXCLUDED. Even two matching labels are only candidates: the system first verifies that they were produced by independent parties, use the same rubric/item generation, contain valid evidence spans, and show no anomalously fast completion or copying fingerprints. Disagreements cannot be averaged into decimal labels; nominal, ordinal, and multilabel tasks each use preregistered rules.

Adjudication Preserves Disagreement Instead of Overwriting History

The dispute packet shows the adjudicator the item, rubric, and two de-identified labels with their evidence, while continuing to hide the model version and experimental winner. The adjudicator produces adjudicationDecision.v1, containing either an accepted label or EXCLUDED_UNRESOLVED, itemized reasoning, cited evidence, whether a rubric gap was found, its scope of impact, and a review deadline. If a gap would change how a class of items is interpreted, rubric v8 should be published first and the affected items explicitly reassigned; the adjudicator must not invent a new standard on the fly under v7.

The sole acceptance point is the label registry’s acceptCanonicalLabel transaction. It simultaneously matches the item generation, rubric digest, independent assignment root, required original label digests, adjudication decision, and a canonical generation that has not yet been set. It then writes an immutable label receipt and atomically switches the item’s active label pointer. Repeated submissions return the same receipt; late labels remain in the evidence ledger but have no authority to overwrite it. An appeal creates generation 2 with supersedes pointing to generation 1. Multi-Agent Result Provenance and Conflict Arbitration provides a general provenance graph; this ledger additionally requires labeling independence and rubric consistency.

An evaluation run must pin a label generation root rather than always reading the active pointer and allowing historical scores to drift. New label generations enter only new runs. If item leakage or systematic mislabeling is discovered, affected old runs are marked EVIDENCE_REVOKED, while the original reports and revocation reasons remain preserved. The immutable membership principle in Training Dataset Snapshot Manifests applies equally to evaluation label sets.

Attacking Consistency with Leakage and Revisions

The failure suite injects duplicate assignments to the same labeler, exposure of the first labeler’s answer to the second, model names leaked through hidden HTML, fixed A/B ordering, item content revisions, rubric upgrades between submissions, evidence spans pointing to old text, duplicate receipts, races between adjudication and late labels, universal abstention on minority classes, and administrators directly editing database labels. Expected results are zero cross-generation acceptances, zero acceptances of non-independent agreement, at most one active-pointer update per generation, and 100% retention of the original disputed judgments.

Gates report Krippendorff alpha, weighted kappa, or per-class agreement rates by label type; they must not use a single aggregate number to conceal minority classes. A production release requires 100% double-label coverage, at least 99.5% valid evidence spans, unresolved disputes below the preregistered limit, adjudication reproducibility on each class’s review sample meeting its target, zero identity/model leakage, and 100% canonical-receipt coverage. First run the new rubric in shadow mode, then dual-label 5% of new items under both tracks and compare their disagreement structures. A rollback only switches the new run’s label-set alias back to the previous complete generation. Existing v8 labels remain sealed and are never overwritten as v7 or deleted because their disputes look unfavorable.