Post-Migration Monitoring

Context

The soak window is the period in which a migration is still reversible, and its length is a deliberate choice rather than a formality: keep the rollback armed for too short a period and the class of fault that surfaces on a weekly cycle arrives after you have stood down; keep it armed indefinitely and nobody is really watching by day four. Seventy-two hours is the common compromise, and what happens inside it decides whether a fault is caught while reverting is cheap or discovered weeks later as an unexplained decline. This work operates the thresholds defined in Rollback Trigger Thresholds and feeds the reversal procedures in Migration Rollback Playbooks.

Monitoring a migration differs from ordinary operational monitoring in one important way: the baseline moved. Every dashboard the team is used to reading was calibrated against the legacy estate, and after a cutover some of its numbers legitimately change — a different origin has different latency characteristics, a new caching layer changes hit rates, a restructured URL set changes the shape of the 404 log. Distinguishing a legitimate shift from a regression is the whole skill, and it depends on having recorded what the old numbers were.

The other distinguishing feature is that the most expensive migration faults are silent. Availability monitoring catches an origin falling over; it does not catch a checkout that returns 200 while failing to charge, a search page rendering with zero results, or a form posting into nothing. Those need signals derived from business outcomes rather than from infrastructure health, and they need somebody looking at them.

Which signals catch which migration faults Five classes of post-migration fault matched to the signal that detects each, showing that infrastructure monitoring alone misses three of them. Infrastructure monitoring catches two of these five Fault Detected by In infra monitoring? Origin failing 5xx rate yes Origin saturating p95 latency yes Redirect gaps 404 log, frequency-ranked no Checkout returning 200 and failing conversion ratio vs last week no Analytics tag not firing a real transaction, by hand no The bottom three are the expensive ones, and none of them registers as an error anywhere in the stack.
A soak that watches only the top two rows will report a healthy migration that is quietly losing revenue.

Pre-flight Checks

None of the items below can be assembled usefully once the window has opened, and several of them describe the legacy estate and stop being obtainable at cutover. Everything below has to exist before the window opens, because none of it can be built usefully during one.

  • Baselines stored as figures — error rate, p95 latency, sessions, conversions, 404 volume — for the same weekday and hour.
  • Alert routing confirmed to reach a pager rather than a dashboard, and tested by firing a real alert.
  • A synthetic probe running from outside your own network against the critical path, on a short interval.
  • The watch rota agreed with named people and explicit handovers, per Building a 72-Hour Migration Watch Rota.
  • Log shipping working from the new origin, verified by finding a request you made yourself.
  • The intentional-retirement list to hand, so expected 404s can be separated from unexpected ones.

Soak Readiness Checklist:

Execution Steps

1. Watch Business Outcomes, Not Only Infrastructure

This is the signal most likely to be missing and the one most likely to fire first on a genuinely bad migration, because it is the only one that measures whether the site is doing its job rather than whether it is responding. Put a conversion or revenue signal on the same screen as the error rate, compared against the same hour of the previous week. This is the trigger that fires first on a genuinely bad migration and the one most likely to be missing, because it belongs to a different team and a different tool. The specific failure modes are covered in Detecting Silent Conversion Failures Post-Migration.

Express the outcome signal as a ratio against the same hour of the previous week, not as a count and not against the previous hour. Counts move with traffic, and an hour-on-hour comparison during a cutover scheduled at a natural trough reports a dramatic collapse that is entirely seasonal. The week-on-week ratio removes both effects and produces a number that means the same thing at 03:00 as it does at midday.

Watch for the signal breaking rather than the outcome. A migration that drops the analytics snippet, changes a tracking identifier, or trips consent handling produces an instant and total collapse in reported conversions while the site works perfectly — so the first response to a conversion trigger is always to confirm the tag is firing, before treating the number as real.

2. Triage the 404 Log Continuously

The 404 log after a cutover is the fastest signal available about redirect coverage, and it is noisy by default. Sort by frequency, diff against the intentional-retirement list, and work the unexplained top entries — a single mis-scoped rule produces thousands of identical failures and sits unmissably at the top. The method is in Log-Based 404 Triage After a Cutover.

Sample the probe path from the visitor’s perspective rather than from a health endpoint. A /healthz returning 200 confirms the process is listening and nothing more, which means a probe built on it will report perfect health while the checkout is failing. Point the probe at a real page on the critical path, and assert on something in the response body as well as the status — a page that renders an error message with a 200 is a common and otherwise invisible outcome.

3. Segment Every Metric by Route Class

An aggregate error rate spread across static pages is a different situation from the same rate concentrated on checkout, and the aggregate hides exactly the case worth catching. Break every metric down by the value tiers established during the audit so a small volume of high-value failures cannot be averaged away by a large volume of healthy traffic.

Expect the 404 log to be noisy and plan for it. A healthy site carries a steady background of requests for paths that never existed — vulnerability scans, malformed URLs, links from other sites with typos — and a migration adds the intentional retirements on top. Ranking by frequency and subtracting the retirement list turns an undifferentiated pile into a short list where a genuine defect is usually the top entry.

Re-run the triage on a schedule rather than once. Redirect gaps surface as crawlers and users work through the long-lived parts of the estate, which happens over days rather than in the first hour, so a single pass on cutover night catches the obvious cases and misses the archive.

4. Keep One Probe With No Shared Dependencies

Your metrics pipeline aggregates through infrastructure the migration may have touched, so a fault that degrades the origin can also degrade the reporting that would have told you about it — and missing data renders as a calm, flat graph rather than an alarm. A dumb loop hitting the real public URL from outside your network, writing somewhere unrelated, is the check to believe when the dashboards disagree with it.

Break out at least the revenue path, the authenticated area, and everything else. Those three behave differently under a migration and fail for different reasons, and a single aggregate number across them is dominated by whichever has the most traffic — which is almost never the one that matters most. Where the audit produced value tiers, reuse them directly rather than inventing a second segmentation.

5. Hand Over Deliberately Between Shifts

Most missed regressions in a long soak happen in the hour after a shift change, when the incoming engineer has the dashboards but not the context that a metric has been drifting upward all night. Require a written handover of three things: what has moved, what has not, and what would make you call it.

How attention decays across a 72-hour window The four phases of a soak window, the fault classes most likely to surface in each, and how the watch should be staffed as attention naturally declines. Staff the window against the decay, not against the clock Phase Surfaces Staffing Hour 0–2 routing, TLS, origin capacity everyone, actively Hour 2–12 caching, session, integration faults one watcher, alert-driven Hour 12–48 batch jobs, scheduled reports rota with handovers Hour 48–72 weekly-cycle and crawl effects one watcher, low cadence The final row carries the highest risk of a missed regression, because the faults are subtle and the attention is lowest.
Tighten the automatic thresholds for the later phases rather than relying on somebody noticing a drift.

Configs / Commands

The fragments below cover the three signals that matter most during a soak: outcome, routing, and an independent liveness check.

Conversions this hour against the same hour last week:

-- A ratio, not a count: traffic varies, the ratio should not
SELECT SUM(CASE WHEN ts >= NOW() - INTERVAL '1 hour' THEN 1 ELSE 0 END)::float
     / NULLIF(SUM(CASE WHEN ts BETWEEN NOW() - INTERVAL '8 days'
                              AND NOW() - INTERVAL '8 days' + INTERVAL '1 hour'
                       THEN 1 ELSE 0 END), 0) AS ratio_vs_last_week
FROM conversions;

Top unexplained 404s, excluding intentional retirements:

# Frequency-ranked 404 paths minus the ones we meant to retire
awk '$9 == 404 {print $7}' access.log | sort | uniq -c | sort -rn \
  | grep -vFf intentional-410s.txt | head -20

Independent synthetic probe with no shared infrastructure:

# Runs off-network, writes to its own log, alerts on three consecutive failures
fails=0
while true; do
  code=$(curl -s -o /dev/null -m 10 -w '%{http_code}' https://www.example.com/checkout)
  [ "${code:0:1}" = "2" ] && fails=0 || fails=$((fails+1))
  [ "$fails" -ge 3 ] && echo "PAGE: checkout $code" >> /var/log/probe.log && break
  sleep 60
done
Escalating from a signal to a decision A metric breaching its threshold, corroborated against the independent probe and the value-tier breakdown, then routed either to a fix-forward or to the reversal depending on whether the fault is bounded. Corroborate, classify, then decide once Threshold breach with its full window Corroborate probe + tier breakdown Classify bounded or not Revert or fix forward one decision Corroboration takes seconds and prevents the most common false abort: a metrics pipeline problem misread as a site problem. A bounded, understood fault on a narrow route is usually faster to fix forward; anything unbounded reverts immediately.
The classification step is written into the runbook precisely so nobody has to invent a taxonomy at three in the morning.

Validation

These verify the monitoring rather than the migration — a soak run on instrumentation nobody has tested reports confidence rather than health. Confirm the monitoring itself works before relying on what it reports.

Write the handover as three fixed lines rather than a free-form summary, because a consistent shape can be read in seconds at the start of a shift and a narrative cannot. Post it in the channel rather than delivering it verbally, so the person after next inherits it too — a chain of written handovers is also the clearest record of how the window unfolded when the post-mortem asks.

Rollback Triggers

These are the thresholds the soak exists to watch; the mechanics of the reversal are in Migration Rollback Playbooks.

  • 5xx above 2% of requests sustained for five minutes, or above 5% for two minutes.
  • p95 latency above 1500 ms, or baseline plus 50%, sustained for five minutes.
  • Conversion ratio against the same hour last week below 0.85 for thirty minutes.
  • Unexplained 404 volume above 1% of requests after the intentional-retirement list is excluded.
  • The independent probe failing three consecutive checks while dashboards report health.

One organisational point that decides whether any of this works: the watcher must have the authority to invoke the rollback without convening anybody. Every mechanism on this page exists to put a number in front of a person quickly, and all of it is wasted if that person then has to find someone more senior to agree. Name the person per shift, state in writing that invoking a documented trigger is always the correct action, and make it clear that the post-mortem examines the threshold’s calibration rather than the caller’s judgement.

FAQ

How much of this can be reused for the next migration? Nearly all of it, and that is the strongest argument for building it properly the first time. The threshold definitions, the probe, the 404 triage script, the outcome query and the handover template are all site-specific rather than migration-specific, so a second migration starts with working instrumentation instead of a blank page. Keep them in the migration repository alongside the audit tooling rather than in somebody’s home directory, and the marginal cost of the next soak window is close to zero.

Why 72 hours rather than 24? Because several important fault classes only surface on a cycle longer than a day: a nightly batch job, a scheduled report, a weekday traffic pattern that differs from the weekend, a cache that only turns over every few hours. Twenty-four hours catches the loud failures and stands down before the quiet ones arrive. Where the business has a weekly rhythm, extending to a full week for the softer signals is reasonable even if the tight infrastructure thresholds relax after three days.

Should the whole team watch, or one person? One person at a time, with a rota and explicit handovers. A shared watch is a watch nobody owns, and the specific failure is that everybody assumes somebody else is looking at the metric that is drifting. Everyone else’s role during the window is to be reachable and to feed the watcher numbers, not to monitor independently.

What if the metrics themselves changed because of the migration? Expect it, and record which ones. A new origin has different latency characteristics, a new caching layer changes hit rates, and a restructured URL set changes the shape of the 404 log — none of which is a regression. The way to keep this manageable is to note the expected shifts before the window and to compare against the stored baseline rather than against yesterday, so a legitimate step change is visible as one rather than being mistaken for a trend.

How do we avoid alert fatigue over three days? Route only the automatic triggers to a pager and everything else to a channel the watcher is actively reading. Seventy-two hours is long enough that a stream of borderline notifications trains people to ignore it, which defeats the exercise entirely. Suppress alerts during known planned events — a batch import, a marketing send — by annotating those windows in advance rather than by loosening the thresholds that cover them.

When is the soak actually over? When a named person says so, on a stated date, after a final check against the baselines — not when everyone drifts away. Closing the window explicitly matters because it is also when the staged rollback assets are retired, the shortened TTL is restored, and the temporary cache headers are reverted. A soak that ends by attrition leaves all of those in place indefinitely.

What should be recorded during the window, and where? The readings at each cadence, the timestamp of every phase transition, and any action taken — all in one place, appended as you go rather than reconstructed afterwards. This costs a few seconds each time and is the only source that can answer the questions a retrospective actually asks: when did the metric start moving, what had just been deployed, and how long was it between the breach and the response. Caches turn over and dashboards roll their retention, so a log written during the window is frequently the only surviving evidence a fortnight later.

Does the soak apply to a phased rollout as well? Yes, per phase, and that is the main reason phasing takes so long in calendar terms. Each phase gets its own window and its own thresholds, because each is exposing a new slice of traffic to the new estate and the faults it surfaces may not have appeared in the previous slice. The compensation is that each window is lower-risk than a single big-bang soak, and that lessons from one phase tighten the monitoring for the next.

Related

← Back to Migration Rollback Playbooks

Explore Sub-topics

Building a 72-Hour Migration Watch Rota
Staff a soak window so one named person is always watching, handovers carry context, and the authority to roll back never needs to be located under pressure.
Log-Based 404 Triage After a Cutover
Turn a noisy post-migration 404 log into a short ranked list of real redirect gaps by subtracting intentional retirements and grouping by pattern.
Detecting Silent Conversion Failures Post-Migration
Catch the migration faults that return HTTP 200 and still lose money — a checkout that never charges, a form that posts into nothing, a search that returns zero results.