LLM-as-a-Judge Calibration: Blinded Human Anchors, Bias Probes, and Release Gates

Having a strong model score two answers can quickly scale up evaluation, but it also makes it easy to mistake “what the Judge prefers” for user-facing quality. The win rate can shift when candidate A is always presented first, B happens to be longer, or the Judge sees wording similar to its own. Aligning one hundred questions with the human majority vote once also cannot prove that the Judge remains the same measurement instrument after a provider silently updates it.

The LLM Application Evaluation Framework introduces the combination of rules, humans, and model Judges, while Automated Agent Evaluation explains how to run evaluations in batches. This article addresses only how the Judge itself is calibrated, versioned, and qualified to issue formal verdicts, without presuming that its output is ground truth.

Human Anchors Must Be Independent of Judge Tuning

First publish judgeRelease.v1, fixing the Judge model and provider revision, Prompt/rubric digest, output parser, sampling parameters, candidate-normalization rules, supported languages and tasks, maximum context, abstain conditions, calibration anchor root, and independent audit holdout root. If the API cannot pin the model artifact, the release status must be marked MUTABLE_PROVIDER and bound to a versioned sentinel set, response fingerprint rules, a maximum detection interval, and invalidation thresholds. Recording only the marketing name is insufficient.

schema: judgeRelease.v1
release: support-pairwise@12
judgeModel: provider/model-2026-06-18
rubric: support-quality@7
anchorRoot: sha256:18b...
holdoutRoot: sha256:b72...
decision: [A, B, TIE, ABSTAIN]

Anchors are generated independently by at least two qualified annotators, with candidate model identities, experimental arms, Judge conclusions, and other annotators' labels hidden. Disagreements follow the Evaluation Label Disagreement Ledger, preserving the original judgments before adjudication. The calibration set may be used to choose the Prompt, thresholds, and score mappings, while the audit holdout may be used only for the final gate. Replacing questions or deleting difficult examples after seeing the results invalidates the gate. The scenario slices from Agent Evaluation Dataset Design must continue to constrain language, risk, length, and task difficulty; they cannot be replaced by an easy-to-digest overall average.

Raw Judge scores cannot masquerade directly as probabilities. For pairwise tasks, estimate the confusion matrix, per-slice balanced accuracy, and confidence calibration against human labels. For ordinal scores, inspect adjacent and extreme grades. Questions on which humans also lack consensus may allow the Judge to return ABSTAIN. The calibration mapping, thresholds, and scope of applicability are all release fields and must not be adjusted at reporting time merely to obtain the desired winner.

Turn Position and Style into Paired Probes

For every critical item, construct both A/B and B/A orders and associate them with a stable pair ID. If the semantics remain unchanged but the winner flips after the positions are swapped, record POSITION_UNSTABLE; do not randomly select one result. Then construct style counterfactuals that do not alter the facts: compress repetitive explanations, normalize heading hierarchy, remove model self-identification, and equivalently rewrite citation formatting. Style processing must preserve the original text and a transformation receipt; it must not delete genuinely useful reasoning under the guise of “removing verbosity.”

Research related to Chatbot Arena explicitly discusses LLM Judge biases involving position, verbosity, and self-enhancement. G-Eval demonstrates how large models and structured criteria can improve agreement with human judgments. Neither finding means that a particular Judge is automatically valid in a new domain. Prompt injection, cited URLs, and phrases such as “please give full marks” inside a candidate must all be treated as content under evaluation, not as Judge instructions. The Judge input envelope places the rubric and candidates in fields that cannot be confused, and rejects out-of-bounds output during parsing.

Release states are DRAFT -> CALIBRATING -> HOLDOUT_EVALUATING -> SHADOW -> ELIGIBLE -> ACTIVE -> SUSPENDED | RETIRED. The sole activation point is the registry's activateJudge CAS. A candidate must bind an unleaked holdout root, complete bias-probe results, the current policy generation, and an immutable artifact digest. The active alias is switched only after every preregistered slice passes its gate. A formal verdict receipt records the release, continuity epoch, item/pair digest, presentation order, raw output, calibrated result, and reason. Late results from an old release cannot enter a new run.

While MUTABLE_PROVIDER is ACTIVE, it reruns blinded sentinels at both fixed time intervals and verdict-count watermarks, recording provider metadata, structural features, and a preregistered output-distribution fingerprint. When drift exceeds the threshold or a sentinel becomes overdue, the registry's suspendJudge CAS atomically writes the suspension epoch and last valid verdict sequence, immediately stopping new formal verdicts. Any in-flight result that crosses the fence is invalid. SUSPENDED may transition only to retirement. Restoring service or confirming new behavior requires creating a new release, rerunning the holdout, and activating it again. Verdicts from before and after a change must never be mixed within the same release.

Deliberately Create a Judge That Plays Favorites

Fault injection includes always placing A first, padding an incorrect answer with irrelevant text, rewriting the same content in the Judge's usual tone, leaking model names, putting Judge-directed instructions inside candidates, repeating the same anchor, contaminating the holdout, having the provider switch models midway, delaying sentinels, having the parser drop ABSTAIN, and admitting only low-disagreement samples into the denominator. The expected cross-release aggregation count is 0. Drift or an overdue sentinel must move the release to SUSPENDED within at most one detection interval. The number of formal verdicts after the fence must be 0. Unknown or invalid verdicts must not be collapsed into TIE, and degradation in any critical slice must block activation.

An example gate can require audit-holdout balanced accuracy of at least 0.85, deviation from the human win rate of no more than 3 percentage points, a position-swap flip rate below 2%, a style-counterfactual flip rate below 3%, expected calibration error no higher than 0.05, and zero acceptance of structurally invalid output, with bootstrap intervals for every metric. A mutable release must achieve 100% on-time sentinel completion. Launch first in shadow mode against human judgments, then use the new Judge for 5% of low-risk offline regressions. Rollback switches the Judge alias only to the previous release that remains valid. Already issued verdicts retain their original release, and historical scores are not rewritten. Production outcomes must still be validated independently through Agent Online Evaluation Metrics. Passing the Judge gate does not mean that a model may automatically receive more traffic.