Blue-Green, Canary, and Rolling Deployment

Many people's understanding of "deployment" stops at npm run build and then tossing the artifact onto a server. This "direct replacement" is practically a ticking time bomb in production—you don't know whether the new version has problems until it has taken over all traffic and all users. Genuinely mature engineering practice cares about a different set of questions: How do you push? How do you cut over? When something goes wrong, how do you return to a safe state in the shortest possible time?

These three questions correspond to three classic deployment strategies: blue-green, canary, and rolling. They aren't mutually exclusive opposites; they're tools under different risk/cost tradeoffs. Below I'll explain each thoroughly, then tell you how to actually choose in production.

1. Deployment Strategy Determines Risk, Not the Version Number

First, establish a notion: the essence of deployment is traffic cutover, not file upload.

A new version pushed up—whether it has problems—99% of the time you can't detect in advance. No matter how high your unit test coverage, you can't reproduce the long tail of real traffic, the boundaries of real data distribution, or the combinations of real dependencies. So "how the new version is exposed to traffic" directly determines:

• Blast radius: When something breaks, how many users/how much traffic is affected;
• Detection speed: How quickly you can spot anomalies in real metrics;
• Recovery speed: How quickly you can cut traffic back to a safe version.

The "direct replacement" strategy bets all three on the new version—the moment it goes live it takes over 100% of traffic, blast radius = full, detection speed depends on user complaints, and recovery speed depends on re-running CI/CD. This is the most naive and most dangerous approach.

All "advanced" deployment strategies are essentially doing the same thing: breaking the "release" action from a "one-shot deal" into a "controllable, gradual process," giving you the chance to discover problems within a small scope and roll back within a small scope. The difference is only in how you break it up and what it costs.

2. Blue-Green Deployment: Two Complete Environments, Instant Route Switch

Blue-green is the most intuitive "double insurance" line of thinking.

How it works:

• Maintain two fully equivalent environments, one called blue (the current production version) and one called green (the version about to go live).
• First deploy the new version to the green environment. At this moment green takes no external traffic, so you can smoke-test and run integration tests to your heart's content.
• Once validation passes, make a configuration switch at the routing layer (load balancer, DNS, Nginx/reverse proxy) to instantly direct inbound traffic from blue to green.
• Once the switch is done, green becomes the new production and the original blue sits idle, becoming the "standby environment" and rollback target for the next round.

Rollback feels great: If something goes wrong in green, switch the route back to blue—just a few seconds, a single configuration change. This is blue-green's biggest advantage: rollback is O(1), clean, and requires no rebuild, because the old version in the blue environment is still fully alive.

The cost is also clear:

1. Double resources. You need two sets of servers/instances/memory, with one sitting idle year-round. For cloud teams on pay-as-you-go, the bill doubles; for resource-strapped small teams, it's often simply unaffordable.
2. The database is the real difficulty. The two application environments can be independent, but there's usually only one database. If the new version ships a schema change:
   • After switching to green the schema changed; rolling back to blue, blue may no longer be compatible with the new schema;
   • So blue-green requires database changes to be forward and backward compatible, or use the "expand-contract" pattern to release across two deployments. This is the pitfall novices hit most easily—assuming blue-green can roll back in seconds, only to have the database lock down the rollback path.
3. Stateful/sessions. If the app holds user sessions in memory, the moment routing switches, green can't see blue's sessions and users get kicked out of their logins. Either externalize sessions (Redis) or cut over during off-peak hours.

Best for:

• Physical machine / VM environments, traditional deployments outside K8s. Blue-green is the cleanest cutover method in such environments.
• Core services with a strong need for "zero downtime + instant rollback."
• Releases where change risk is manageable but you want an extra safety net (e.g., with a compatible schema change alongside).

3. Canary Deployment: Probe with a Small Slice of Traffic, Then Scale Up

The name "canary" comes from old miners—before going down the shaft they'd lower a canary in first; only if the bird was fine would the humans descend. It's the same in deployment: first let the new version take a small slice of real traffic and see whether it "dies."

How it works:

1. The new and old versions are online simultaneously; through weight configuration at the routing layer, the new version receives only a small portion of traffic—say 5%.
2. During the observation window (minutes to hours), keep a close eye on key metrics:
   • Error rate (5xx, exception counts);
   • Latency (P95, P99);
   • Business metrics (conversion rate, payment success rate—these explain the situation better than technical metrics);
   • New anomaly patterns in logs/alerts.
3. Metrics healthy → ramp up to the next tier (e.g., 25% → 50% → 100%);
4. Anomalies at any tier → immediately drop the new version's traffic weight to 0; all traffic returns to the old version and rollback is complete.

Compared to blue-green, canary is "truly gradual"—blue-green is an instant switch between 0% and 100%, whereas canary lets you pause and observe at any ratio in between. This means:

• Smallest blast radius: at most 5% of traffic is affected when something breaks;
• You can make decisions based on real data rather than pre-release test data;
• Naturally suited to A/B testing—split traffic by proportion across versions and compare business outcomes.

Cost:

1. Requires fine-grained traffic routing. An ordinary round-robin load balancer can't split traffic by weight; you need:
   • Service meshes like Istio/Linkerd that switch traffic by weight at the Service Mesh layer;
   • Nginx/Envoy weighted upstream;
   • K8s + Argo Rollouts, which packages "canary + automated metric judgment + automatic rollback" into a declarative CRD—currently the most mainstream approach in the cloud-native world.
2. Monitoring/observability must be in place. Without metrics split by version dimension, canary is blind—you see the overall error rate go up but have no idea whether it's the new version or the old. Prometheus + version labels, distributed tracing, and dashboards aggregated by version are all prerequisites.
3. Observation period. The ramp-up pace can't be too fast; each tier needs ample observation time. This means a single release can take hours or even days. Teams in a hurry to release will find it slow.
4. Version compatibility. With old and new versions online and reading/writing the database simultaneously, the API must be backward compatible (expanded on in the rolling deployment section below).

Best for:

• Large-scale, high-QPS online services, where 5% of traffic is enough to expose the vast majority of problems.
• High-risk changes: major refactors, new dependencies, performance-sensitive modifications.
• K8s / service mesh environments with mature toolchains.

4. Rolling Deployment: K8s's Default Choice, Gradual Replacement

Rolling deployment is the default strategy of the Kubernetes Deployment, and the "everyday release method" for the vast majority of cloud-native services.

How it works:

• Rather than replacing all instances at once, it works in batches:
  • Start several Pods of the new version;
  • Once the new Pods are ready, begin taking down some old Pods;
  • Start more new Pods, take down more old Pods... rolling along until everything is replaced.
• Pace is controlled with two parameters: maxSurge (the upper limit above the desired replica count) and maxUnavailable (the lower limit of replicas allowed to be unavailable). For example, with replica count 10, maxSurge=1, maxUnavailable=0, it means at any time at least 10 are available, at most 11, with new Pods rotating in one at a time.

Advantages:

1. Doesn't require double resources. It only adds "a little surge capacity"; compared to blue-green's full redundancy, the cost is very low. This is the fundamental reason it's the K8s default.
2. No extra routing configuration needed. K8s's Service + Endpoints handles traffic automatically; you just change the image tag.
3. Smooth process. Users perceive no interruption.

Cost (key):

1. Brief coexistence of old and new versions. During rolling, the cluster has both v1 and v2 Pods at the same time. This means:
   • The API must be backward compatible. v2 must be able to read data written by v1 and handle requests sent by v1; conversely, v1 must be compatible with v2. Otherwise cross-version calls will fail.
   • Database schema changes demand extra care. Again, expand-contract: first add the field (both v1/v2 can read/write it) → deploy v2 and switch to the new field → then clean up the old field. Never do a destructive schema change directly during rolling.
2. Rollback is slower than blue-green. When something breaks and you need to roll back, it's effectively another rolling deployment in reverse—swapping v2 back to v1 one at a time. Not as slow as a rebuild, but not an instant switch either.
3. No "observation window". Rolling advances at a steady pace, without canary's "pause and observe" mechanism. Unless you manually pause, it rolls all the way to the end.

Best for:

• Stateless services + forward/backward compatible versions—this is rolling's comfort zone.
• Everyday iterative releases where change risk isn't high and there are no destructive schema changes.

5. Feature Flags: Decouple "Deployment" from "Release"

Having covered the three strategies, we must add another dimension: Feature Flags. They're the best partner for any deployment strategy; the core idea is—

Code deployment ≠ feature release.

How it works in practice: deploy the new code as usual (rolling or canary), but the new feature is wrapped in a flag, defaulting to off. Production runs the new code, but users don't see the new feature—this is called a dark launch. When you're ready, flip the switch in your config center (LaunchDarkly, Unleash, a self-built Apollo/Nacos) and the feature becomes visible to users; if something goes wrong, turn the flag off, no code rollback needed.

Feature flags solve several problems that deployment strategies alone can't:

1. Separating release from launch. Code can be deployed days in advance; features are toggled to the business's cadence (coordinated with marketing campaigns, gray-launch cities).
2. Instant kill, no redeploy needed. Rolling back code takes minutes to hours (rebuild + release); turning off a flag takes seconds.
3. Granular ramp-up by dimension. Flags can do fine-grained gray launches by user ID, region, or whitelist—more flexible than canary's "by traffic percentage."
4. Reduces the pressure of deployment itself. Deployment just swaps code; risk is compressed to the "can it even run" layer.

Note: flags are technical debt. Once a feature is fully live and stable, clean up the flag and dead code promptly; otherwise your codebase will pile up with layers of if feature_flag_xxx and become a maintenance nightmare.

In practice, canary + feature flags is the most robust combination today: canary handles "gradual ramp-up at the code level," and feature flags handle "instant toggle at the feature level." Layered together, they essentially minimize release risk.

6. Production Selection: No Silver Bullet—Depends on Environment and Change

The three strategies each have tradeoffs; how do you actually choose in production? Here's a practical decision framework.

Strategy Comparison

Choose by Environment

• K8s environments:
  • Use rolling by default—it's out-of-the-box and covers 90% of everyday releases.
  • High-risk changes / want fine-grained ramp-up → go with canary, using Argo Rollouts (declarative, auto-judges by metrics, auto-rollback) or Istio traffic weights. This is the de facto standard in the cloud-native world.
  • You generally wouldn't force blue-green inside K8s—rolling + canary already covers the problems blue-green solves, and more resource-efficiently.
• Physical machine / VM environments:
  • Without K8s's Pod concept, rolling replacement is costly → blue-green is the cleanest approach. Maintain two groups of machines and switch upstreams in Nginx/HAProxy.
  • If resources are genuinely tight → at least do "half blue-green": blue-green only for core services, rolling for the surrounding services.

Choose by Change Type

• Routine iteration, no schema change: rolling.
• Major refactor, new dependency, performance change: canary, run on a small slice of traffic first.
• Destructive change (incompatible API/data): don't use any "gradual" strategy—first split it into multiple compatible releases with expand-contract, then rolling/canary. Forcing a rolling deploy will just make you cry during rollback.
• Marketing campaigns / need business rhythm control: feature flags, hand the launch timing to the business.

7. Rollback Is a First-Class Citizen

Finally, the most easily overlooked principle:

Before deploying, first figure out how to roll back.

This isn't chicken soup; it's engineering discipline. Many teams spend 90% of their deployment plan on "how to go up," and only one line on "how to go back"—"roll back if there's a problem"—only to discover when something actually breaks that:

• Rollback requires rebuilding the image, and CI/CD takes 20 minutes; during those 20 minutes production keeps bleeding;
• The database has already written the new version's data/format; after rolling back to the old version, the old code can't read it;
• The new version depended on a new downstream service; after rollback the old version doesn't call it, but the downstream is actually already using caches in the new format...

Treating rollback as a first-class citizen means:

1. The rollback path must be verified before deployment. Blue-green: confirm the old environment is still there and the route can be switched back. Canary: confirm the weight can be instantly zeroed. Rolling: confirm the old image tag is still there and a reverse rolling deploy is possible.
2. Database changes must be designed to be reversible. The expand-contract pattern: add a field (compatible) → deploy a dual-write version → switch reads → clean up the old field. Any irreversible DDL (such as a direct DROP COLUMN) blocks rollback.
3. Rollbacks must be rehearsed. Chaos Engineering / Game Day should include a "forced rollback" drill so the rollback action becomes muscle memory, rather than the first real run being an actual incident.
4. Monitoring metrics must be able to trigger automatic rollback. Tools like Argo Rollouts can integrate with Prometheus to automatically roll back when error rate/latency cross thresholds, compressing MTTD+MTTR to the minute level. Having a human watch monitoring and roll back manually often means a response time of tens of minutes.

For deployment, the measure of maturity isn't "how fast you can go live" but "how fast you can return to a safe state when something breaks." Put rollback first and releases actually become more relaxed.

Summary

1. The essence of deployment is traffic cutover, not file upload. What determines risk isn't the version number but "how you push, how you switch, how you roll back."
2. Blue-green: instant switch between two environments, extremely fast rollback; the cost is double resources. Suited to physical machines/VMs and strong rollback needs, but database compatibility is the biggest pitfall.
3. Canary: probe with small traffic then ramp up, smallest blast radius; the cost is reliance on fine-grained routing and mature monitoring. In K8s, land it with Argo Rollouts / Istio.
4. Rolling: K8s default, low cost and smooth process; the cost is brief old/new coexistence and a must for forward/backward compatibility.
5. Feature flags: decouple "deployment" from "release"—code can be deployed in advance and features toggled instantly; the best partner for every strategy (but also technical debt, to be cleaned up regularly).
6. Rollback is a first-class citizen: figure out the rollback path before deploying, design database changes to be reversible, and let key metrics trigger automatic rollback.

Bottom line:

A good deployment isn't "successfully shipping a new version"; it's "no matter whether the new version is good or bad, I can return to a safe state in the shortest possible time." The yardstick of maturity has never been launch speed—it's rollback speed.