Inference Reproducibility Across Hardware and Runtimes: From Artifact Snapshots to Statistical Equivalence Gates

The same model, the same prompt, and the same seed can still produce different answers on two GPUs. Floating-point addition order, fused kernels, cuBLAS or cuDNN algorithms, quantization implementations, tensor-parallel reductions, dynamic batching, and random-number consumption order can all introduce tiny differences in logits. When two tokens have nearly equal scores, those differences can amplify throughout autoregressive generation. Treating the lack of bitwise consistency as normal is not enough. A platform must still define which level of reproducibility it can guarantee and prove that upgrades have not changed business outcomes.

Start by Defining Four Levels of Reproducibility

The first level is artifact reproducibility: given a record, you can retrieve exactly the same weights, tokenizer, adapter, chat template, and runtime image. The second is environment reproducibility: the hardware SKU, driver, CUDA, cuBLAS, cuDNN, compiler, kernel selection, and parallel topology are identical. The third is token reproducibility: the same encoded input and decoding parameters produce the same output token IDs. The fourth is statistical equivalence: when nondeterministic kernels or different hardware are allowed, quality, probability distributions, and performance on a fixed evaluation set remain within predefined ranges.

Different use cases require different targets. Incident forensics and unit tests usually require the first three levels. An A100-to-H100 migration is unlikely to guarantee bitwise consistency, but it must satisfy the fourth level. A report that merely says the outputs are "roughly the same" supports neither replay nor any judgment about whether the next difference is larger. The LLM Evaluation Framework provides task metrics; this approach adds environmental and numerical evidence.

Snapshot Every Input That Affects Logits

Request records should include at least the model and tokenizer digests, input token IDs, attention mask, adapter, quantization configuration, chat template, seed, temperature, top-p, top-k, maximum output length, stop tokens, logit processor order, and output token IDs. Environment records should include the container digest, GPU name and compute capability, driver, CUDA/cuBLAS/cuDNN versions, inference framework commit, compilation options, tensor/pipeline parallelism, batch size, the request's position within the batch, and whether CUDA Graph is enabled.

{
  "runId": "replay-2026-07-18-042",
  "model": "sha256:75bf...",
  "runtime": "sha256:b901...",
  "gpu": "H100-SXM-80GB",
  "driver": "575.57.08",
  "cuda": "12.9",
  "seed": 41872,
  "sampling": { "temperature": 0, "topP": 1 },
  "scheduler": { "maxNumSeqs": 64, "tensorParallel": 2 }
}

Saving only the prompt text is insufficient because normalization or tokenizer changes can produce a different sequence of IDs. Saving only the seed is also insufficient because concurrent requests can change the order in which random numbers are consumed. Each request should use an independent random stream derived from the global seed and a stable request ID, rather than sharing a single global generator that advances with each batch. Replayable data can be organized using Agent Replay Evaluation as a reference, while sensitive inputs should be governed by access controls and retention periods.

Deterministic Mode Has Explicit Costs

In PyTorch, you can enable deterministic algorithms and disable benchmark mode, which may select different algorithms. Some CUDA configurations may also require CUBLAS_WORKSPACE_CONFIG=:4096:8. These settings can reduce throughput and still do not guarantee complete consistency across releases or platforms. If an operator has no deterministic implementation, the test should fail and identify the operator rather than silently falling back. Whether production enables these settings depends on the risk profile, but the test environment must be able to reproduce the release report.

Even when temperature is zero, greedy decoding is not necessarily consistent across hardware: reduction errors can reverse the order of nearly tied logits. Therefore, record the top-k tokens and logprob differences at every step. If the top two tokens at the first output divergence differ by only 1e-6, that is not the same class of failure as a large distribution shift caused by incorrect input encoding.

Build a hardware x driver x runtime x precision x parallelism matrix containing the combinations that production will actually run. Freeze a sample set covering short text, long contexts, Chinese, code, structured output, cases near stop boundaries, and known low-margin tokens. First run token consistency tests in single-request mode without batching. Then run dynamic batching with real request-arrival sequences to distinguish numerical kernel effects from scheduling effects.

Replace Vague Tolerance with Equivalence Intervals

For audit tasks that require exact token-by-token consistency, any token divergence should block the release. For open-ended responses, freeze statistical gates in advance. For example: critical policy test cases must pass 100%; task accuracy must not fall by more than 0.5 percentage points relative to the previous environment; structured-output success rates must not decline; the maximum absolute logit error before the first divergence must remain below an agreed threshold; and output lengths and refusal rates across 1,000 fixed samples must remain within their historical 95% intervals. For sampling tasks, run multiple independent seeds for every input and compare distributions rather than drawing conclusions from a single output.

Performance is also part of the environment contract. At the same quality level, record time to first token, per-token latency, throughput, GPU memory usage, and error rate. If disabling high-performance kernels for determinism makes the SLO unattainable, explicitly choose statistical equivalence mode instead of quietly restoring nondeterministic settings. Production monitoring can integrate with LLM SLO Monitoring and group results by environment digest so that averaging across old and new replicas does not conceal drift.

Before an upgrade, run shadow replays and then route a fixed tenant cohort into the new matrix. If a critical policy test fails, a statistical interval is breached, the first divergence occurs substantially earlier, or performance exceeds its threshold, route traffic back to the previous container digest and hardware pool. Retain the raw token evidence from both environments. The release and recovery process can use Canary Rollouts for Model Serving, but the previous drivers, images, and capacity must remain genuinely available throughout the observation window.

Reliable reproducibility does not mean claiming that every GPU will produce bitwise-identical results. It means stating precisely which combinations produce the same tokens, which can guarantee only statistical equivalence, and ensuring that every difference can be traced to specific artifacts, environments, scheduling behavior, and numerical evidence.

Further Reading

• LLM Evaluation Framework
• Agent Replay Evaluation
• LLM SLO Monitoring
• Canary Rollouts for Model Serving