Canary Rollouts for Model Serving: Make Traffic Shifts Verifiable and Rollback-Ready

Changing the model image from v17 to v18 and gradually increasing its traffic weight after the Pod becomes ready may look identical to a canary rollout for an ordinary web service. The actual risks are more complex: the model weights may load successfully while response quality declines, a new tokenizer may change input-length accounting, a quantization kernel may produce numerical anomalies on certain GPU models, an unwarmed KV Cache may cause the first users to time out, or the same session may drift between old and new versions and lose context. A model-serving canary rollout must validate both infrastructure and model behavior.

The release unit should be an immutable "service version" that includes the model weight digest, tokenizer, inference runtime and its image digest, quantization method, GPU architecture, default sampling parameters, system prompt version, and safety policy version. Recording only a model name is not enough to support rollback; even with identical weights, changes to the runtime or default parameters can alter the output. Once the build artifact enters the artifact registry, it must be signed, and the release record must be linked to the offline evaluation report.

All Four Gates Are Mandatory

The first is the artifact and compatibility gate: verify hashes, licenses, image vulnerabilities, GPU driver and CUDA compatibility, then run deterministic smoke tests after startup. The second is the offline quality gate: compare task success rates, structured outputs, citation correctness, safe refusals, and critical slices on a frozen dataset. A global average must not be allowed to hide regressions in a particular language or high-risk task. The sample stratification described in the LLM Evaluation Framework is suitable for codifying as a release contract.

The third is the performance gate: load-test with production-like distributions of input length, output length, and concurrency, recording queueing time, time to first token, per-token latency, throughput, peak VRAM usage, and OOM events. The fourth is the production rollout gate: verify that the old version remains available, routing supports one-click rollback, capacity can handle all traffic after rollback, and observability labels distinguish between versions. Without evidence for every gate, the release controller must not begin shifting traffic.

Use Shadow Requests to Find Problems Before They Affect Users

During the shadow stage, copy a small portion of redacted requests that are approved for evaluation to the new version without returning its responses to users. Do not copy requests involving write-capable tools, sensitive original content, or calls that cannot be executed idempotently. When necessary, send only replay samples with similar characteristics. Compare error categories, output formats, latency, token usage, and rule-based scores between the old and new versions, and conduct blind reviews of high-value samples.

Shadow traffic cannot prove the real user experience because it does not face real queueing and may not include complete sessions. After passing the shadow gate, begin the canary with internal accounts or explicitly designated low-risk tenants. Recommended stages are 1%, 5%, 20%, 50%, and 100%. Each stage must cover at least one business peak or a predefined minimum request count, whichever provides stronger evidence. Automatically promoting a stage when its timer expires can let a dozen successful requests during a low-traffic period substitute for meaningful evidence.

steps:
  - setWeight: 1
  - analysis: {template: model-quality-and-slo, minRequests: 1000}
  - setWeight: 5
  - pause: {duration: 30m}
  - setWeight: 20
  - analysis: {template: model-quality-and-slo, minRequests: 5000}

The thresholds in this configuration must come from the current stable version's baseline and business SLOs, not from copied example values. Use automated gates for error rate, P95 time to first token, OOM events, GPU utilization, and per-request cost. For semantic quality, use quickly computable task signals, user retry rates, format validation, and sampled reviews. The metric system can be aligned with LLM SLO Monitoring.

Handle Sessions, Caches, and Warm-Up

Multi-turn sessions should remain sticky within a compatible version family, or their session state should be migrated explicitly. If the new version changes the tokenizer, tool protocol, or context format, switching midway may prevent historical messages from being reused. The routing key should include the session ID and release version. During rollback, define which sessions remain on the new version and which return to the old version from the next turn onward, and ensure that clients do not splice together streamed output from two versions.

Prompt caches and KV Cache entries must be tagged with the model weights, tokenizer, and inference parameter versions. The new version must not read caches created by the old version. Before shifting real traffic, use representative requests without sensitive content to load weights, compile kernels, and warm up critical input lengths. Successful warm-up does not mean the service is ready; reserved VRAM and concurrency slots must also be confirmed. The Kubernetes readiness probe must not succeed until the service can actually process requests, rather than as soon as the port starts listening.

AI Feature Flags can bind model versions to tenants and tasks instead of routing only by a random site-wide percentage. If a regression is found in a particular coding task, its rollout can be stopped while other validated scenarios continue expanding.

Automatic Rollback Must Cover Semantics and Capacity

Conditions for immediate rollback include bypassed safety policies, cross-tenant data exposure, sustained OOM events, a surge in invalid response formats, or abnormal side effects from tool calls. Conditions that should pause the rollout for human review include a slight decline in proxy quality metrics, increased tokens per request, or tail latency approaching its budget. Every condition must define its evaluation window, minimum sample size, and behavior when data is missing. If monitoring loses visibility, pause by default instead of treating "no metrics" as healthy.

The rollback process must first set the new version's traffic weight to zero, then wait for in-flight streaming requests to finish or reach their deadline, and finally retain the new instances and logs for diagnosis. The old version should maintain warm capacity for at least the observation period. If it is immediately scaled to zero to save GPU capacity, so-called automatic rollback becomes a lengthy process of pulling the weights again. Capacity planning must assume that the canary and old version will coexist temporarily and verify that the old pool can reclaim all traffic during a failure.

Do not delete the old weights and configuration immediately after the rollout completes. Retain the release manifest, metric snapshots from each stage, human approvals, rollback events, and sample differences until the observation window ends. If metrics improve because answers are shorter but incomplete, the postmortem must explain quality and cost together instead of celebrating only the latency reduction.

Prove Control-Plane Reliability Through Drills

In a preproduction environment, deliberately make the analysis metrics API time out, make the new version's readiness flap, trigger GPU OOM events, change the output Schema, and initiate a rollback at the 20% stage. Verify that the controller stops promotion, routing weights actually converge, alerts identify the specific version, the old service restores capacity, and session streams are not interleaved. The release process itself must also fall within the authorization, approval, and audit boundaries of CI/CD Pipeline Design, preventing anyone from bypassing the gates and directly modifying production traffic weights.

The success criterion for a model canary is not that "the new Pod is running." It is that the new version has used enough samples from clearly defined cohorts and tasks to prove that quality, performance, safety, and cost all remain within budget, while every step can quickly return to a known stable state.

Further Reading

• LLM Evaluation Framework
• LLM SLO Monitoring
• AI Feature Flags
• CI/CD Pipeline Design