Quantized Model Regression Gates: Quality, Performance, and Rollback Thresholds

A quantized model may start successfully and use less GPU memory while quietly degrading on a small number of high-value requests: an extra character appears at a JSON boundary, Chinese proper names are substituted, evidence near the end of a long context is ignored, or a safety refusal becomes a direct answer. Comparing only the average score on a public benchmark neither proves that production tasks remain correct nor answers whether the saved GPU memory justifies the regression. Quantization should be treated as the release of a new model artifact, not as a runtime parameter that can be switched temporarily.

Freeze the Candidate Identity First

Generate an immutable manifest before every test. At minimum, record the base-weight digest, quantization algorithm and implementation, bit width, group size, calibration-data hash, layers excluded from quantization, weight and activation data types, inference runtime, kernel version, driver, and GPU model. model=foo-int4 is not an auditable identity; once a directory with the same name is re-exported, previous reports immediately lose their meaning.

candidate: sha256:8e1c...
baseModel: sha256:72ab...
quantizer: awq-0.2.9
weightBits: 4
groupSize: 128
calibrationSet: sha256:34fd...
runtime: vllm-0.10.1
gpu: NVIDIA-H100-80GB

The calibration and regression sets must remain separate. The former determines scale or clipping values, while the latter evaluates generalization. Reusing the same samples for both biases the results toward a distribution the model has already seen. The artifact manifest must also bind the tokenizer and chat template so that the quantized candidate does not carry another undeclared change. The versioning principles from the LLM Evaluation Framework can be reused to construct the general evaluation set, but numerical-precision-sensitive slices must be added here.

Release Based on Slices, Not the Overall Average

Establish two controls: the floating-point baseline and the current production version. The floating-point baseline shows how much quantization itself degrades performance, while the current production version shows what changes users will perceive. The regression set should reflect the actual mix of production tasks while also providing separate slices for Chinese, code, mathematics, structured output, extremely long inputs, repetitive text, rare proper names, refusals, and tool arguments. Critical compliance cases require zero tolerance and cannot be offset by high scores on a large number of simple samples.

An executable gate might require all 180 safety and authorization cases in a fixed regression set of 2,000 samples to pass; overall accuracy on primary tasks to decline by no more than 0.7 percentage points relative to the current version; no slice representing more than 5% of traffic to decline by more than 2 percentage points; JSON Schema validity not to decrease; and zero occurrences of NaN, garbled text, or infinite repetition in the output. Thresholds should come from business losses and historical variance, not be copied from this example.

For open-ended generation, do not rely only on exact string matches. Depending on the task, combine rule-based validation, blinded human evaluation, model judges, and pairwise preferences, while retaining the original input, token IDs, output, scorer version, and failure reason. Replay the same frozen set for every candidate, then supplement it with recent production failures. This feedback loop can integrate with Evaluation-Driven Development, preventing the gates from permanently covering only the problems known at release time.

Performance Gains Must Also Clear the Gate

The absence of a significant quality regression only shows that the candidate is "harmless"; it does not show that quantization is "valuable." Using identical hardware, runtime parameters, and request arrival sequences, measure static GPU memory usage, available KV Cache, time to first token at P50/P95/P99, decoding tokens/s, maximum stable concurrency, power consumption, and cost per million tokens. Run a mix of long and short requests for at least 30 minutes, including cancellations, bursts, and cold cache starts. Peak performance for a single request does not represent production gains.

For example, a release contract could require weight memory usage to decrease by at least 35%, stable throughput under the same latency SLO to improve by at least 20%, P95 time to first token to degrade by no more than 10%, and the error rate not to increase. If INT4 saves GPU memory but reduces throughput because the target GPU lacks a suitable kernel, reject the candidate. Continue using the actual token distribution described in GPU Inference Capacity Planning for capacity calculations instead of extrapolating from short laboratory prompts.

Quantization support often depends on specific hardware capabilities. Run a smoke-test and stress-test matrix across every GPU SKU, driver, and tensor-parallel topology used in production, checking for fallback to slow kernels, OOM errors, numerical overflow, and output drift between replicas. Combinations that have not been covered should be prohibited at the scheduling layer rather than labeled "theoretically compatible."

Use the Same Artifact Boundary for Canary Releases and Rollbacks

After the offline gates pass, first shadow sanitized requests, then route 1%, 5%, and 20% of traffic by tenant or a stable hash. In production, monitor task success rates, retries, refusals, structured-output parsing, token length, time to first token, decoding speed, and GPU memory levels. The observation window must cover at least one real traffic peak; a five-minute health check cannot reveal long-tail regressions. Canary Releases for Model Serving can provide traffic control, while the quantization gates define when to proceed or stop.

Rollback does not mean re-exporting a "fixed" version. The platform must retain the validated floating-point or previous quantized digest, its corresponding tokenizer, runtime image, and sufficient spare capacity. If any critical case fails, a primary slice crosses its threshold, or P95 latency exceeds its limit for three consecutive five-minute windows, immediately route traffic back to the old digest and prevent new sessions from entering the candidate. Depending on risk, drain or terminate existing sessions. After rollback, retain candidate request samples and configuration snapshots, distinguish among calibration problems, kernel problems, and changes in data distribution, and then generate a new candidate identifier.

The final output of a quantization gate is not a screenshot claiming that "accuracy dropped by only 1%." It is a reproducible body of evidence: a precisely identified artifact, frozen slices, differences against two baselines, gains on target hardware, production stop conditions, and a tested recovery path to the previous version.

Further Reading

• LLM Evaluation Framework
• Evaluation-Driven Development
• GPU Inference Capacity Planning
• Canary Releases for Model Serving