Building a 72-Hour Migration Watch Rota

Problem Statement

A soak window fails through staffing far more often than through instrumentation. The dashboards are usually fine; what goes wrong is that hour forty is covered by nobody in particular, the person who takes over at 02:00 has the graphs but not the knowledge that error rate has been creeping up all evening, and the one person empowered to call the rollback is asleep with their phone on silent. Seventy-two hours is long enough that goodwill will not cover it, so it has to be a rota with names, shifts, handovers and delegated authority. This page sits under Post-Migration Monitoring.

When to Use This Approach

  • The soak window spans at least one overnight period.
  • The team is small enough that “everyone watches” would mean nobody does.
  • Rollback authority currently sits with one person who cannot be awake for three days.
  • Previous migrations have produced regressions found the following week.
  • Stakeholders expect status updates on a predictable cadence.

Step-by-Step Instructions

1. Divide the Window Into Named Shifts

Write the rota as a table of shift, start, end, watcher and deputy, and publish it before the window opens. Six to eight hour shifts work; twelve is too long for anyone to stay attentive to a slow drift, and four generates more handovers than it saves. Every shift needs a named individual rather than a team, because a shift owned by a team is a shift owned by nobody.

Publish the rota where the people on it will look, which is rarely the document it was authored in. A pinned message in the channel the team already uses beats a comprehensive plan in a shared drive, because the question it answers — whose shift is it, and who is my deputy — is asked at odd hours by somebody half awake.

2. Delegate Rollback Authority to Each Shift

The watcher on duty must be able to invoke the reversal without waking anybody to ask. Rollback value decays fast — the difference between calling it at minute five and minute forty is often the difference between a non-event and a recovery project — so a rota whose authority is centralised has an escalation delay built into every hour of the night. State the delegation in writing, and state explicitly that invoking a documented trigger is always correct even when the fault later proves minor.

3. Make the Handover a Written Artefact

Require three lines at every shift change: what has moved, what has not, and what would make you call it. Post it in the channel rather than delivering it verbally, so the shift after next inherits it too. Most missed regressions in a long soak happen in the hour after a handover, when the incoming watcher has the dashboards but not the trend.

HANDOVER 02:00 — from A to B
Moved:      5xx 0.3% -> 0.6% since 22:00, all on /search. p95 flat.
Not moved:  conversion ratio 0.98, 404 volume steady, probe green.
Would call: 5xx above 2% for 5 min, or any /checkout non-200.

Have each watcher perform the reversal dry-run themselves before the window rather than accepting that the team has tested it. Credentials expire per person, permissions differ per role, and the specific failure this catches is a watcher who discovers at 04:00 that their account lacks the CDN purge permission the runbook assumes. Ten minutes each, before the window, removes an entire class of stall.

4. Set a Cadence for Status, Separate From Alerts

Alerts interrupt; status reassures. Publish a short update on a fixed schedule — every four hours works for a 72-hour window — containing the same four items each time: phase, readings against thresholds, next decision point, time of next update. Sending it when nothing has changed is the point: silence is read as trouble and generates exactly the interruptions the cadence exists to prevent.

5. Close the Window Explicitly

Name the person who declares the soak over and the date they will do it. Closing is not just a communication — it is when the staged rollback assets are retired, the shortened TTL restored, the temporary cache headers reverted and the monitoring returned to normal thresholds. A soak that ends by attrition leaves all of those in place indefinitely.

What each shift needs before the window opens Five attributes every shift in a watch rota requires — a named watcher, a deputy, delegated authority, working credentials and a handover — with the failure that follows if each is missing. Five things per shift, and each absence has a signature failure Attribute If missing A named individual the shift is owned by nobody in particular A named deputy one silent phone and the alert goes nowhere Delegated rollback authority an escalation delay built into every night hour Working credentials, tested the reversal stalls on a login at 04:00 A written handover the incoming watcher has graphs but no trend The last row is the one most often skipped and the one that made the difference in the worked example below.
None of these can be arranged during the window — they are all pre-flight items with names attached.

Worked Example

A team of five runs a Thursday-evening cutover with a 72-hour window. They divide it into eleven shifts of six to eight hours, with the two engineers who built the migration taking the first and the last, and a deputy named for every shift.

At 03:40 on Saturday — hour 32, the shift nobody wanted — the watcher sees the conversion ratio drop to 0.71 and holds for twenty minutes before it crosses the thirty-minute window. The handover note from the previous shift said the ratio had been sitting at 0.94 rather than the usual 0.99, which was the drift that made the drop legible rather than ambiguous:

HANDOVER 02:00 — from C to D
Moved:      conversion ratio 0.99 -> 0.94 over the last three hours. Nothing else.
Not moved:  5xx 0.2%, p95 340ms, probe green, 404 volume flat.
Would call: ratio below 0.85 for 30 min.

They invoke the rollback at 04:12 without calling anybody, because the delegation was written down. The cause turns out to be a payment provider rejecting requests from the new origin’s IP range — a fault with no error signature on the site at all. Without the drift recorded in the handover, the 03:40 reading would have looked like ordinary overnight noise.

A handover that carries the trend, not just the state A written handover recording what has moved, what has not, and the threshold that would prompt a call, so the incoming watcher inherits the direction of travel rather than a snapshot. Three lines, the same three every time What has moved with the numbers What has not so silence is explicit What would make you call the threshold, restated Posted in channel not spoken A snapshot tells the incoming watcher where things are; the trend tells them whether that is good news. Posting rather than speaking means the shift after next inherits the same context, and the post-mortem inherits all of it.
The fixed shape matters more than the detail — a consistent format is read in seconds, a narrative is skimmed.

Set the reading cadence explicitly rather than leaving it to judgement. Every fifteen or twenty minutes is sustainable across a six-hour shift and frequent enough to see a drift developing; continuous watching is neither, and a watcher who has been staring at graphs since midnight is less reliable by hour four rather than more.

Verification

Confirm the rota is real before the window rather than during it.

# 1. Every hour of the window has exactly one named watcher
awk -F, 'NR>1 {for (h=$2; h<$3; h++) c[h]++} END {for (h=0; h<72; h++) if (c[h]!=1) print "hour", h, "->", c[h]+0}' rota.csv

# 2. Every watcher can actually reach the alerting channel
# (fire a test alert per shift owner and require an acknowledgement)

# 3. Every watcher has working rollback credentials
#    — have each one run the reversal dry-run against staging before the window

A clean result is exactly one watcher per hour, an acknowledged test alert from every shift owner, and a completed dry-run per person rather than per team.

Alerts versus the human cadence The two monitoring mechanisms during a soak compared by what each is for, what it catches, and why neither substitutes for the other. Alerts interrupt; the human cadence notices drift Mechanism Catches Cannot catch Automatic alerts a threshold breach, immediately a slow approach to one Watcher reading every 20 min drift below any threshold a sudden spike between reads Status update every 4h nothing — it reassures anything at all Independent probe a pipeline failure masking both subtle degradation The second row is the one this rota exists to staff, and it is the only mechanism that sees a metric moving before it breaches.
The third row is not monitoring and should never be counted as such — its job is to prevent interruptions, not to detect faults.

FAQ

Is a 72-hour rota realistic for a small team? For a team of four or five, yes, at roughly two shifts each — which is demanding but bounded. Below that, shorten the window rather than pretending to cover it: a genuinely watched 36 hours is worth considerably more than a nominal 72 in which half the shifts are unstaffed. What you must not do is publish a rota with names against hours nobody has agreed to.

What should the watcher actually do for six hours? Read the dashboard at a fixed cadence — every fifteen or twenty minutes is plenty — and do something else in between. A watcher staring at graphs continuously will be exhausted by hour three and less reliable, not more. The alerts exist to interrupt for anything urgent; the human cadence exists to catch the slow drift that no threshold has fired on yet, which is what the handover note records.

Should the people who built the migration take the difficult shifts? They should take the first and the last, and share the rest. The opening shift benefits most from deep familiarity because that is when routing and infrastructure faults surface, and the closing shift is where the decision to stand down is made. Spreading the overnight shifts across the wider team is both fairer and better for the organisation, since it distributes knowledge of how the estate behaves.

How do we handle a fault that spans a handover? The outgoing watcher stays until it is resolved or explicitly transferred with context — a handover mid-incident is the single most dangerous moment in the window. Where the incident is long-running, do the transfer deliberately: walk the incoming watcher through what has been observed and what has been tried, and record it in the channel rather than only in conversation.

Does the rota need a deputy for every shift? Yes, and the deputy needs the same credentials and the same delegated authority rather than being a name on a list. The failure this prevents is specific and common: the watcher’s phone is on silent, or they are unreachable for twenty minutes, and the alert goes nowhere. A deputy who cannot actually perform the reversal is not cover.

Related

← Back to Post-Migration Monitoring