Reproducible AI Evaluation Run Artifacts: A Closed Manifest from Dataset to Evaluator

A report that reads "dataset v4, model-x, temperature 0" may still be irreproducible a month later: v4's labels have been revised, the model alias points to a new snapshot, the Prompt is read from remote config, the batch order changed the sampling, and the Evaluator has upgraded its parser. The final 82.4 score is reduced to a single unverifiable number, and the team cannot tell whether the difference comes from the candidate model or the evaluation infrastructure.

Cross-hardware inference reproducibility distinguishes per-token consistency from statistical equivalence; this article extends the scope to the input, execution, and verdict closure of an entire evaluation run. Training dataset snapshot manifest and lineage governs training data selection; evaluation must additionally pin down gold label generation, sample order, and the evaluator.

First seal evaluationRunManifest.v1

The manifest at minimum contains the run ID, the purpose and pre-registered metrics, the dataset membership root, the label root, the split/query, the model weight digest or vendor revision, the tokenizer, the Prompt/context bundle, the tools and retrieval fixtures, all decoder parameters, seed semantics, batch size and concurrency, the harness commit, the Evaluator/Judge release, the parser and disposition schema, the runner image, the dependency lock, the hardware/kernel, the locale/timezone, and the external-service policy. For secrets, record only controlled references and versions, never plaintext.

schema: evaluationRunManifest.v1
run: eval-2026q3-184
datasetRoot: sha256:2de...
labelRoot: sha256:11a...
model: weights@sha256:9c3...
promptBundle: support@sha256:58f...
decoder: {temperature: 0, topP: 1, maxTokens: 2048}
runnerImage: sha256:4ab...
evaluator: judge-release@12
reproducibilityClass: EXACT_REQUEST_STATISTICAL_OUTPUT

The OCI Image Specification provides the foundation for content-addressed artifacts and manifests, but it cannot enumerate the semantic dependencies on behalf of an evaluation system; W3C PROV's entity/activity/agent relationships can connect dataset, run, response, and verdict, but they also do not automatically guarantee membership integrity. The manifest seals digests before dispatch, and any field change creates a new run. When a vendor only offers a mutable model name, the search hits the live web, or a tool must access production data, explicitly mark it EXTERNAL_MUTABLE, save each raw observation receipt, and lower the reproducibility class — never fabricate a weight digest.

Prompt versioning and CI handles Prompt publishing; this run records the exact bundle. LLM Context Dispatch pins the actual message sequence sent to the model. Both the template digest and the final sequence must be saved, because truncation, the ordering of tool descriptions, or dynamic dates can make the same template produce different requests. A fixed seed is also not a per-token guarantee across vendors, GPUs, and batches; it can only act as part of the complete decoding contract.

Every expected item must have a unique destination

The run state machine is DRAFT -> SEALED -> DISPATCHING -> COLLECTING -> RECONCILING -> VERIFIED | INCOMPLETE | INVALID; once missing evidence is filled in, INCOMPLETE can only return explicitly to RECONCILING. Build a unique execution slot for (runId, itemId, sampleOrdinal), and writes of retried attempts go into the evidence list of the same slot. When a response is lost, query by the provider request ID; if it cannot be confirmed, record OUTCOME_UNKNOWN, and you must not switch the ID, rerun, and leave behind only the successful samples. Rejected-by-policy, timeout, parse failure, and Judge abstain are all mutually exclusive terminal states and enter the pre-registered handling of the metrics denominator; OUTCOME_UNKNOWN can be used for reconciliation but always blocks the run from reaching VERIFIED.

The collector computes the expected membership root, the terminal disposition root, the raw request/response root, the trace root, and the evaluator result root, and verifies that expected = scored + rejected + timeout + invalid + abstained + unknown + excluded_by_plan. This equation only proves denominator conservation; as long as unknown > 0, a shard is missing, an item is duplicated, or the evaluator mixes generations, the run stays in INCOMPLETE. Cost, latency, and failure are also attributed by the same item identity and cannot be estimated from another round's log.

The only acceptance point is the evidence registry's acceptEvaluationRun CAS: it accepts only runs whose state is VERIFIED and unknown = 0, and simultaneously matches the sealed manifest, the expected root, all dispositions, the Evaluator release, the environment attestation, the reconciliation conservation, and the unset acceptance slot, and then writes the run receipt at the FORMAL level. The reporting layer reads only that formal receipt; a diagnostic snapshot of INCOMPLETE must be tagged PROVISIONAL and must not enter the model release gate or the final score. Dashboards must not query "the latest partial results" and pass them off as the formal receipt. Agent replay evaluation can consume this artifact to review control flow, but cannot replace the original run evidence.

Start breakage from a single missing field and silent upgrades

The fault suite injects a mutable dataset URL, a label pointer changed version, a model alias switched mid-run, a prompt config cached across generations, batch reordering, a seed ignored by the SDK, runner image tag drift, region/locale differences, a permanently missing shard, duplicate callbacks, output truncation, a Judge upgrade, and a trace upload failure. Assert that runs with a missing field or containing UNKNOWN are formally accepted at 0, that cross-digest results merge at 0, that each expected slot has exactly one terminal state, that all UNKNOWN remain queryable, and that all diagnostic reports are tagged provisional.

The release gate requires manifest/disposition required-field coverage, membership reconciliation, and receipt lineage all at 100%, that UNKNOWN for formal runs is 0, that the rate of consistent request digests rebuilt from a clean runner is 100%; a local small model declared deterministic must match output item by item, otherwise it is accepted against the pre-registered statistical distance and confidence interval; external mutable dependencies must not enter the EXACT class. A new harness first shadow-generates the evidence root against the same sealed run before taking over 5% of daily evaluation. On an anomaly, switch the runner alias back to the previous artifact and create a new run, rather than stitching old and new attempts together; all sealed manifests, failed terminal states, and reports are retained so that rolling back the executor does not roll back historical fact.