Setting Latency Ceilings That Trigger Rollback

Problem Statement

Error rates make an easy rollback trigger because a 500 is unambiguous. Latency is harder: the new stack is slower, but slower than what, measured where, at which percentile, and for how long before it counts? Without those four answers written down before the cutover, the conversation at 03:00 becomes a negotiation, and the migration either rolls back on a two-minute blip or limps on through a genuine regression because nobody can say the threshold was crossed. This page sits under Rollback Trigger Thresholds.

Four decisions a latency trigger must specify Percentile, measurement point, comparison baseline and sustained duration, each shown as a component of a complete rollback trigger definition. A Trigger Is Four Decisions, Not a Number Percentile p75 and p95 never the mean Measured where edge, not origin what users feel Against what same hour last week not an absolute ms For how long 10 min sustained the anti-blip rule Leave any one of the four unstated and the trigger cannot be evaluated under pressure. Write them as a single sentence the on-call engineer can read and act on.
A complete trigger reads: p75 edge latency more than 1.5× the same hour last week, sustained ten minutes.

When to Use This Approach

  • You are writing rollback criteria for a cutover and error rate alone is not sufficient.
  • The new stack has a different architecture — a new CDN, server-side rendering where there was none, a different database — so absolute latency numbers are not comparable.
  • The business cares about conversion, which degrades with latency long before anything errors.
  • Previous cutovers produced arguments about whether performance was “bad enough” to revert.

Why Latency Triggers Get Argued About

An error-rate trigger works because the underlying signal is categorical: a response either failed or it did not, and counting failures needs no interpretation. Latency is continuous, multi-modal and context-dependent, so every part of the measurement is a choice, and any choice left unstated becomes a point of disagreement at the moment you can least afford one.

The disagreements are predictable. Somebody quotes the mean, which is lower and calmer than what users experience. Somebody else quotes p99, which is dominated by bots and retries and is alarming at all times. A third person points out that the measurement is taken at the origin, so it excludes the extra network hop the migration just introduced. A fourth notes that the comparison is against yesterday, which was a bank holiday. All four are making reasonable points, and collectively they guarantee that no decision is made.

Writing the four choices down in advance does not make the measurement more accurate. It makes it decidable, which is the property a rollback trigger actually needs. A trigger that is slightly wrong but unambiguous is far more useful at 03:00 than one that is theoretically ideal and open to interpretation.

# Publish the exact command that produces the trigger number — no interpretation left over
curl -s -o /dev/null -w '%{time_starttransfer}\n' https://example.com/p/example-product

There is one further reason to settle this before the cutover: the baseline has to be captured while the legacy stack is still serving. After the switch, the number you needed to compare against no longer exists.

Step-by-Step Instructions

1. Capture a Baseline You Can Compare Against

A latency trigger is meaningless without the number it is a multiple of. Capture at least a full week of the legacy stack at the same percentiles and the same measurement point you will use afterwards — the discipline described in Capturing Core Web Vitals Baselines Before Migration.

# Per-hour p75 and p95 from the access log, in milliseconds
awk '{print substr($4,14,2), $NF*1000}' /var/log/nginx/access.log \
  | sort -k1,1n -k2,2n \
  | awk '{h[$1]=h[$1]" "$2} END {for (k in h) {n=split(h[k],v," ");
      print k, "p75="int(v[int(n*0.75)]), "p95="int(v[int(n*0.95)])}}' | sort -n

2. Choose Percentiles That Represent Users, Not Servers

Use p75 as the primary trigger and p95 as the secondary. The mean hides everything that matters, and p99 is dominated by bots, retries and pathological requests — it moves for reasons that have nothing to do with your migration.

3. Measure at the Edge

Origin timings exclude TLS negotiation, CDN cache misses, and the extra hop a migration often introduces. Measure where the user is, and separate the two so a regression can be attributed.

# Split the timing so you can attribute a regression to connect, TLS, or the app
curl -s -o /dev/null -w \
  'dns=%{time_namelookup} connect=%{time_connect} tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total}\n' \
  https://example.com/p/example-product

4. Require a Sustained Breach

Every cutover produces a cold-cache spike in the first minutes. Require the ceiling to be exceeded continuously for a defined window — ten minutes is a reasonable default — so the trigger cannot fire on a warm-up artefact.

Sustained breach versus a cold-cache spike Two latency traces above a ceiling line: a short warm-up spike that returns below the ceiling, and a sustained regression that stays above it for the full window. Only One of These Should Trigger ceiling warm-up spike sustained regression t+0 t+20 min t+45 min The green trace crosses the ceiling first and is the one you must not act on.
Peak height is a poor discriminator; time above the ceiling separates the two cases cleanly.

5. Pair the Ceiling With a Named Decision Owner

A threshold with no owner is a discussion. Name the role that calls it, and give that person the authority to invoke the rollback path without further escalation — the same structure used for the error-rate triggers in Defining Error-Rate Thresholds That Trigger Rollback.

Worked Example

A media site moved from a static-file origin behind one CDN to a server-rendered application behind another. The baseline week gave p75 time-to-first-byte of 180 ms and p95 of 410 ms at the edge, measured hourly.

The trigger written into the runbook was:

Roll back if p75 TTFB at the edge exceeds 270 ms (1.5× baseline) for ten consecutive minutes, or if p95 exceeds 1,200 ms (roughly 3× baseline) for five consecutive minutes. Called by the on-call SRE without further escalation.

On cutover night p75 hit 640 ms for the first four minutes — cold caches on a new edge network — and settled at 210 ms. No trigger fired, correctly. At hour six a deployment of the recommendation service pushed p95 to 1,900 ms and held it there; the five-minute secondary trigger fired at minute six and the service was rolled back on its own, without reverting the migration.

The distinction mattered: the latency trigger identified a component, and the response was proportionate. A single undifferentiated “site is slow” rule would have reverted the whole cutover for a fault in one service.

Verification

# 1. Confirm the measurement point produces the numbers the trigger assumes
for i in $(seq 1 20); do
  curl -s -o /dev/null -w '%{time_starttransfer}\n' https://example.com/
done | sort -n | awk '{v[NR]=$1} END {print "p75="v[int(NR*0.75)], "p95="v[int(NR*0.95)]}'

# 2. Baseline and current are measured the same way — same path, same region, same cache state
curl -s -o /dev/null -w 'cache=%{header_json}\n' -I https://example.com/ | grep -io 'cf-cache-status[^,]*'

# 3. The alerting rule actually fires — test it against replayed data, not in production
promtool test rules latency-rollback-tests.yaml

Check three is the one teams skip and regret. A rollback trigger that has never fired in a test is an assumption, and cutover night is a poor time to discover that the query returns no data because the new stack reports its metric under a different label.

Latency budget split across the request path A stacked bar dividing total time into DNS, connect, TLS, origin processing and transfer, with the migration-sensitive segments highlighted. Where a Migration Adds Milliseconds DNS connect TLS origin processing transfer new resolver new edge PoP new chain new application — usually here unchanged Splitting the timing tells you whether to roll back the migration or fix one component. A trigger on total time alone cannot make that distinction.
Attribution is what converts a latency alert into a proportionate response rather than a full revert.

FAQ

Should the ceiling be an absolute number or a multiple of baseline? A multiple, with an absolute floor. A multiple travels across regions and devices where an absolute number does not; the floor stops a very fast baseline making a trivial absolute increase look catastrophic.

Which percentile should trigger a rollback? p75 as the primary signal and p95 as a faster secondary with a higher multiplier. The mean is useless because it hides bimodal behaviour, and p99 is noisy enough to fire on traffic patterns unrelated to the migration.

How long should a breach persist before it counts? Long enough to exclude cache warm-up — ten minutes for the primary trigger, five for a severe secondary. Instantaneous triggers fire on every cutover, which teaches on-call engineers to ignore them.

Do Core Web Vitals belong in a rollback trigger? They belong in the post-migration review rather than in the live trigger. Field vitals aggregate over 28 days, so they cannot tell you anything on cutover night; server-side and synthetic timings are what you act on in the moment.

What if latency is worse but conversion is unchanged? Record it, do not roll back for it, and fix it forward. The trigger exists to protect against harm, and a regression with no measurable business effect is a performance ticket rather than an incident — provided the measurement window is long enough to be trusted.

Should the trigger cover every page template or just the important ones? Measure a small named set — home, a category, a product or article, and the checkout or conversion path — rather than an average across everything. A site-wide aggregate hides a catastrophic regression on one template behind good numbers everywhere else.

How do I set a ceiling when the new stack is architecturally different? Use the multiple-of-baseline form and accept that the absolute number will move. A server-rendered page replacing static files will legitimately be slower at the origin and may be no slower at the edge; measuring where the user is keeps the comparison meaningful across the change.

Does a cold cache justify a higher ceiling for the first hour? It justifies the sustained-breach window, which is a cleaner mechanism than a time-varying threshold. A ceiling that changes on a schedule is another thing to get wrong at 03:00; a ceiling that must be exceeded for ten minutes handles warm-up without special cases.

Who should own the latency trigger — SRE or the business? SRE owns calling it; the business owns agreeing the number beforehand. Splitting it that way means the threshold reflects what the organisation is prepared to tolerate and the decision at the moment is not a negotiation.

What should happen when a trigger fires but the cause is one component? Roll back that component, not the migration. The trigger’s job is to demand action within a time bound, not to prescribe the widest possible action — which is why the request-path breakdown matters more than the total.

Related

← Back to Rollback Trigger Thresholds