Implementing Blue-Green Deployments for Site Migrations

Problem Statement

A single-environment migration forces you to break the old site to build the new one, so any defect surfaces in production with no clean way back — your only option is a forward fix under pressure while users see errors. DNS propagation delays split traffic across environments, stale CDN caches serve broken assets, and database writes can land in both places during the overlap. A blue-green deployment removes the all-or-nothing risk: you stand up the new environment (green) alongside the live one (blue), validate strict parity, then shift traffic gradually with an instant revert path back to blue. The discipline that makes it work is twofold — green must be a genuine clone (config, data, headers, session keys), and the rollback must be automated against numeric thresholds so the decision to revert is not a panicked judgement call at 2am. This page sits under Zero-Downtime Cutover Plans and covers building the two environments to strict parity, shifting traffic with a canary stage, and wiring rollback triggers that fire automatically.

Blue-green traffic shift A router sends live traffic to the blue environment, then weighted-shifts to green; a rollback path returns traffic to blue on failure. Blue-Green Traffic Shift DNS / LB router weighted routing Blue (current) legacy production Green (new) migration target rollback on 5xx > 2%
The router weight-shifts traffic from blue to green; breaching an error or latency threshold returns it to blue instantly.

When to Use This Approach

  • Downtime is unacceptable and you need an instant revert path rather than a forward-fix.
  • You can afford to run two parallel environments for the duration of the cutover.
  • Your routing layer (DNS, load balancer, or CDN) supports weighted traffic shifting.
  • Database and session state can be kept consistent across both environments during overlap.
  • You want automated, threshold-driven rollback instead of a manual judgement call under pressure.
  • You have observability (APM or log analysis) able to compute 5xx rate and p95 latency in near real time, since those numbers drive the rollback decision.

Step-by-Step Instructions

1. Establish Infrastructure Parity

Green must match blue before any traffic moves. Define both with infrastructure-as-code and diff the rendered config to catch drift in server blocks, headers, and routing.

# Fail fast on any difference between live and target server config
diff -rq /etc/nginx/sites-available/ /staging/etc/nginx/sites-available/
# Tag responses so you can see which environment served a request
# (in the green server block)  proxy_set_header X-Environment green;

2. Synchronise Data and Files

Mirror the database and file system from blue to green, then verify integrity before the switch. For the database integrity gate, follow Syncing Staging Databases Before Production Switch.

# Files: checksum-verified mirror, dry-run first to validate the delta
rsync -avz --checksum --delete --dry-run /var/www/html/ /mnt/green/html/
rsync -avz --checksum --delete         /var/www/html/ /mnt/green/html/
# Confirm identical signing keys / session store so logins survive the switch

3. Pre-Stage DNS and Shift Traffic

Lower TTL ahead of time so the shift — and any revert — propagates in minutes, then move weight from blue to green in stages rather than all at once. Start with a small canary (for example 10%), hold it long enough to observe real traffic against your error and latency budget, then ramp to 50% and 100%. Track adoption with Monitoring Global DNS Propagation During Cutover and hold full cutover until convergence.

# Execute the weighted record change, then watch propagation
aws route53 change-resource-record-sets \
  --hosted-zone-id "$ZONE" --change-batch file://cutover.json   # blue -> green
watch -n 10 'dig +noall +answer example.com @8.8.8.8'           # TTL decays to 60s

4. Purge Caches and Validate Live

Purge the CDN after the flip so the edge re-fetches from green, then validate SEO-critical headers and asset integrity in real time.

# Purge edge, then confirm canonical/robots headers and asset hashes
curl -s -X POST "https://api.cloudflare.com/client/v4/zones/$ZONE/purge_cache" \
  -H "Authorization: Bearer $TOKEN" -d '{"purge_everything": true}'   # full purge
curl -sI https://example.com | grep -iE 'canonical|x-robots-tag'
sha256sum -c manifest.sha256   # every asset matches the green manifest

Parity is the word doing all the work in those four steps, and it is worth being concrete about what has to match. Infrastructure diffing catches the server blocks; the items that actually bite are the ones that live outside the config files.

What has to match between blue and green before traffic moves Six parity items compared across the blue and green environments, showing which are caught by configuration diffing and which are only caught by an explicit check, together with the symptom each produces if it is missed. Config diffing finds three of these — the rest need their own check Must match Caught by Symptom if missed Server blocks / routing config diff 404s on paths that worked yesterday Response headers config diff a missing X-Robots-Tag deindexes pages TLS chain + intermediates config diff browsers fine, mobile apps reject Session signing keys explicit check every user logged out mid-shift Scheduled jobs / crons explicit check both environments run them, twice Outbound source IP explicit check payment provider blocks the new host
The bottom three never appear in a server-config diff, and each produces a failure that looks like an application bug rather than a migration one.

Worked Example

An agency migrates example.com from a legacy VM (blue, 198.51.100.5) to a containerised stack (green, 203.0.113.20). Parity diffing in step 1 catches a missing X-Robots-Tag header on green that would have deindexed paginated pages; they add it before proceeding. After an rsync --checksum mirror and a database checksum match, they lower TTL to 60 s and push a Route 53 weighted record at 90% blue / 10% green.

The 10% canary holds for 15 minutes at a 0.3% 5xx rate and 240 ms p95, so they shift to 50/50, then 100% green. They purge the CDN and confirm the edge serves green:

curl -sI https://example.com/app.js | grep -i 'x-cache-status'
# x-cache-status: MISS   <- edge re-fetched from green after purge

Twenty minutes later an APM alert shows green’s checkout endpoint hitting 3.1% 5xx — above the 2% trigger. The pre-authored rollback.json re-points the record to blue, the CDN is purged again, and error rates fall to baseline within 4 minutes because the TTL was already at 60 s. The migration is retried the next night after fixing the checkout regression.

The rollback in that example took four minutes because every part of it was decided in advance. That is the property worth copying: the threshold, the payload, and the purge were all authored before the window, so the only thing happening under pressure was execution.

Timeline of an automatic threshold-driven revert From the checkout error rate crossing two percent, through alert, pre-authored DNS revert and cache purge, to error rates returning to baseline four minutes later because the TTL was already lowered to sixty seconds. Four minutes, because nothing was decided during them 5xx on /checkout threshold 2.0% Action all pre-authored 0.3% 3.1% — breach falling back to baseline alert rollback.json CDN purge T+0 breach detected T+4 min The 60 s TTL set days earlier is what makes the revert land in minutes rather than hours. A revert that has to be written during the incident is not a rollback plan.
Every box on the lower lane existed before the window opened; the incident consisted only of running them in order.

Verification

Confirm propagation, then confirm green is healthy before trusting it with full traffic.

# 1. All major resolvers agree on the green IP
for r in 8.8.8.8 1.1.1.1 208.67.222.222; do dig @"$r" example.com A +short; done
# 2. Error and latency budget within thresholds (from APM/log analysis)
awk '$9 ~ /^5/ {c++} END {print "5xx:", c+0}' access.log   # expect near zero
# 3. SEO headers and robots parity match blue
curl -sI https://example.com/robots.txt | head -n 1   # expect 200

If thresholds are breached, revert per Rollback Trigger Thresholds rather than attempting a forward fix mid-cutover.

FAQ

How do I verify DNS propagation before committing the blue-green switch? Run dig +noall +answer example.com against several global resolvers (8.8.8.8, 1.1.1.1, 208.67.222.222) and watch the TTL decay to the pre-configured 60 s floor. When every sampled resolver returns the green IP, propagation is effectively complete and you can commit the full shift.

What does the X-Environment header buy me that logs do not? It removes an entire class of ambiguity from every subsequent investigation. During a weighted shift, the single most common question is “which environment served this request?”, and without a response header the answer has to be inferred by correlating timestamps across two log sets — slow when you are calm and unreliable when you are not. Tagging the response makes it a one-line curl, lets support staff attach the environment to a ticket without engineering help, and gives your synthetic monitoring something unambiguous to assert on. Strip the header once blue is decommissioned, or leave it and treat it as permanent deployment metadata.

What is the safest method to sync large media directories without downtime? Use rsync -avz --checksum --delete --bwlimit=5000 (limit in KiB per second) and run a --dry-run first to validate the delta calculation before the real transfer. Verify integrity with sha256sum -c manifest.sha256 against a manifest generated on blue before you cut over.

What triggers an automatic rollback in a blue-green migration? Trigger rollback on a 5xx error rate above 2% over a 5-minute window, p95 latency above 800 ms, or failed /healthz checks on three consecutive polls. The rollback script must execute the DNS revert and CDN purge atomically, and the rollback.json payload must be pre-authored and tested before the cutover window opens.

How long should blue stay running after a successful shift? Long enough to cover at least one full traffic cycle plus every scheduled job that has not yet run on green — commonly three to seven days. Keep it warm and serving, not merely powered on, because a rollback to an environment whose caches are cold and whose connection pools have been idle for a week will itself look like an outage. Disable blue’s cron jobs and outbound integrations the moment green takes full traffic, though, or both environments will keep sending the same emails and writing the same reports.

Does blue-green work when the two environments share one database? Yes, and that is the common case for a hosting migration as opposed to a schema migration — sharing the database is precisely what makes an instant revert safe, because no data diverges. The constraint it imposes is that green’s application code must remain compatible with the current schema in both directions, so any schema change has to be deployed as an additive step first and a cleanup step much later. If green requires a schema that blue cannot read, you no longer have a rollback path, whatever the routing layer says.

Related

← Back to Zero-Downtime Cutover Plans