Defining Error-Rate Thresholds That Trigger Rollback

Problem Statement

“Roll back if errors get bad” is not a plan — under pressure it produces hesitation while error rates climb. You need concrete error-rate numbers, evaluated over fixed windows, that page the right person or auto-fire the reversal. This is the detailed companion to Rollback Trigger Thresholds, covering how to set 5xx and 4xx ceilings, choose windows, and wire alerting that turns a metric into an action.

Error-rate threshold zones A 5xx error-rate scale divided into three zones: healthy below 0.5 percent, warn between 0.5 and 2 percent, and auto-rollback above 2 percent over a five-minute window. 5xx Error-Rate Decision Zones evaluated over a rolling 5-minute window at the edge HEALTHY WARN — page owner AUTO-ROLLBACK < 0.5% 0.5% – 2% > 2% 0% 0.5% 2% 5%+ Baseline 5xx before cutover is typically < 0.1% — set ceilings against measured baseline, not zero.
Three pre-agreed zones turn a live error-rate metric into an unambiguous action: continue, escalate, or revert.

When to Use This Approach

  • You are in the soak window after a cutover and need unambiguous abort criteria.
  • Your monitoring exposes per-status-code request rates at edge and origin.
  • You want certain breaches to auto-fire and others to require owner confirmation.
  • You need to distinguish origin failures (5xx) from broken routing (4xx).
  • Stakeholders demand a defensible, pre-agreed basis for any rollback decision.

Step-by-Step Instructions

1. Separate the 5xx and 4xx Ceilings

Treat the two error classes distinctly: 5xx signals origin/application failure, 4xx signals broken redirects or missing resources. Set a 5xx ratio ceiling (e.g. 2% over 5 minutes) and a separate 4xx ceiling relative to baseline.

# 5xx ratio across all requests, evaluated for a 5-minute window
sum(rate(http_requests_total{code=~"5.."}[5m]))
  / sum(rate(http_requests_total[5m])) > 0.02

Evaluate each class at the edge and at the origin, as separate series. A CDN serving cached copies will happily mask an origin returning 5xx to every uncached request, so the edge series looks healthy right up to the moment the cache expires and the failure arrives all at once. The inverse also happens: an expired certificate or a bad rule at the edge produces errors that never touch the origin and so never appear in application metrics. Two series, two alerts, and a note in the runbook about what a divergence between them means.

2. Choose Windows That Filter Noise but Bound Damage

Pick a window long enough to ignore a cache-warm blip yet short enough to cap exposure. Five minutes suits 5xx; ten minutes suits 4xx, which is noisier from crawlers.

# Prometheus: require the breach to persist for the full window before firing
- alert: Rollback5xxBreach
  expr: sum(rate(http_requests_total{code=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.02
  for: 5m
  labels: {severity: page, action: rollback}

3. Add a Fast Auto-Fire Tier for Severe Breaches

Define a second, stricter tier that auto-triggers without confirmation when the breach is unambiguous, so catastrophic failures do not wait on a human.

# Severe tier: 5xx above 5% for 2 minutes auto-triggers the reversal
- alert: Rollback5xxSevere
  expr: sum(rate(http_requests_total{code=~"5.."}[2m])) / sum(rate(http_requests_total[2m])) > 0.05
  for: 2m
  labels: {severity: critical, action: auto-rollback}

Segment the ceiling by route class as well. An aggregate 2% spread evenly across a large volume of static pages is a very different situation from 2% concentrated entirely in checkout, and the aggregate figure hides precisely the case worth catching — a small number of high-value requests failing completely. At minimum, evaluate the revenue path as its own series with a tighter ceiling than the site-wide one.

4. Wire Alerting to a Pager, Not a Dashboard

Route the breach to the named rollback owner via pager and a synthetic backstop, so detection never depends on someone watching a screen.

# Synthetic backstop: page if the critical path returns 5xx three times in a row
fails=0
while true; do
  c=$(curl -s -o /dev/null -w '%{http_code}' https://www.example.com/checkout)
  [ "${c:0:1}" = "5" ] && fails=$((fails+1)) || fails=0
  [ "$fails" -ge 3 ] && echo "PAGE: sustained 5xx ($c)" && break
  sleep 20
done

Put together, the four steps produce a two-tier ladder rather than a single line: one tier fast and automatic for breaches with no benign explanation, one slower and human for breaches that might be a crawler or a broken analytics tag.

Two-tier error-rate ladder from baseline to auto-rollback Error rate bands from a 0.1 percent baseline through a healthy zone, a paging tier at two percent held for five minutes, and an auto-rollback tier at five percent held for two minutes. Two tiers, because not every breach means the same thing 5xx above 5% held 2 min — auto-rollback, no confirmation sought 5xx above 2% held 5 min — pages the named owner, who decides Baseline 0.1% healthy — transient spikes below the window are ignored no benign explanation could be a crawler or tag no action The shorter window on the severe tier is deliberate: the worse the breach, the less evidence you should require before acting.
Band height is not proportional to error rate — it is proportional to how much deliberation each band deserves.

Worked Example

A retailer cut over its checkout flow at 09:00. Baseline 5xx was 0.1% and 4xx was 1.2%. Thresholds were set at 5xx > 2% for 5 minutes (page), 5xx > 5% for 2 minutes (auto-rollback), and 4xx > baseline + 10 points for 10 minutes (page).

At 09:06 the new origin’s connection pool saturated; 5xx climbed to 3.4% and held. The Rollback5xxBreach alert fired at 09:11 after the full 5-minute window, paging the release owner, who confirmed the abort. Separately, a broken redirect map pushed 4xx from 1.2% to 13%, breaching the 4xx ceiling at 09:16 — this pointed at routing rather than origin, so recovery used Redirect Rollback & Recovery rather than a DNS revert. Had 5xx instead spiked to 6%, Rollback5xxSevere would have auto-fired at 09:08, skipping the confirmation step entirely. The two distinct error classes drove two distinct recovery paths, exactly as the threshold design intended.

The most useful thing in that example is not either threshold firing — it is that the two error classes routed to two different recovery procedures. The 5xx breach pointed at the origin and would have been fixed by a DNS reversion; the 4xx breach pointed at routing and was fixed by a config swap in seconds. Conflating them into a single “error rate” would have sent the team to the slow remedy for the fast problem.

Error class determines which rollback procedure to run A 5xx breach indicates origin or application failure and routes to the TTL-bound DNS reversion, while a 4xx breach indicates broken routing and routes to the atomic redirect swap that recovers in seconds. The status class tells you which runbook to open Error rate breaches its ceiling 5xx — origin failing app, pool, or upstream 4xx — routing broken redirects, missing assets DNS reversion bounded by the TTL Redirect swap seconds, no TTL wait Reaching for a DNS rollback to fix a 4xx problem trades a thirty-second recovery for a TTL-bound one — the most common misdiagnosis in this table.
Both breaches can occur at once, as in the example above. Splitting the alerts is what lets you run both remedies in parallel rather than guessing which fault you are looking at.

Verification

  • Inject a controlled 5xx burst in staging and confirm Rollback5xxBreach fires only after the full 5-minute window, not on a transient spike.
  • Confirm Rollback5xxSevere auto-fires within ~2 minutes on a >5% burst, and that the page reaches the owner’s device.
  • Drive a test path to 404 and confirm the 4xx alert fires while 5xx stays quiet, proving the classes are independent.

FAQ

What is a sensible default 5xx threshold for a migration soak? A ratio of 2% sustained over 5 minutes for the paging tier, plus a 5% over 2 minutes auto-fire tier. These work as absolute floors even without a clean baseline; tighten them toward baseline + a small margin once you have stable pre-migration numbers.

Why evaluate 4xx separately from 5xx? They indicate different failures. A 5xx surge means the origin or application is failing and a DNS or origin rollback is likely; a 4xx surge usually means redirects or resources are broken and the fix lives in the routing layer. Conflating them sends you down the wrong recovery path.

Should every error-rate breach automatically roll back? No. Reserve auto-fire for severe, unambiguous breaches. Moderate breaches should page the named owner for a confirmation call, because softer rises can stem from benign causes like a crawler hitting deprecated paths or analytics tagging breaking during the move.

Should the ceiling be a ratio or an absolute count? A ratio for the threshold, with an absolute floor underneath it. Ratios are what you want because they stay meaningful as traffic varies — 2% is 2% at midnight and at midday. The problem is that at very low volume a ratio becomes unstable: two failed requests out of thirty is 6.7% and means almost nothing. Add a minimum request count to the rule so the ratio is only evaluated once there is enough traffic to be meaningful, or you will spend the quiet hours of the soak window responding to statistical noise.

How do I keep crawler traffic from distorting the 4xx rate? Exclude known crawler user agents from the 4xx calculation, or evaluate a separate series for them. Search engines routinely request paths that have not existed for years, so a healthy site carries a steady background of crawler 404s that has nothing to do with your migration — and during a cutover crawl activity often increases, which can push the aggregate figure past a ceiling calibrated on human traffic. Keep the crawler series visible as its own signal, since a genuine spike in it is exactly the crawl-error trigger, but do not let it drive the routing alarm.

What if the migration changes what a normal error rate looks like? Recalibrate deliberately rather than letting the thresholds drift. A new platform may legitimately return 404 where the old one returned 410, or move an endpoint that a third-party integration still calls. Identify those expected changes before cutover, quantify them, and either exclude them from the series or raise the baseline you compare against — with the reasoning written down. What you must not do is widen the ceiling during the incident because the numbers are higher than expected; that is indistinguishable from ignoring the trigger.

Does the synthetic backstop duplicate the monitoring, and is that wasteful? It duplicates it deliberately, and the duplication is the point. Your metrics pipeline aggregates through infrastructure that the migration may have touched, so a failure that degrades the origin can also degrade the reporting that would have told you about it — and missing data renders as a flat, calm-looking graph rather than as an alarm. A dumb loop hitting the real public URL from outside your network, writing to somewhere unrelated, has almost no shared dependencies with the thing it is watching. When the two disagree, believe the loop.

Related

← Back to Rollback Trigger Thresholds