Detecting Silent Conversion Failures Post-Migration

Problem Statement

The most expensive migration failures return 200. A checkout that renders, accepts a card, shows a confirmation page and never charges anything; a contact form that submits successfully and posts into a queue nobody is consuming; a search page that returns zero results for every query because an index was not migrated. None of these produce an error rate, a latency spike, or a log line that looks wrong — and all of them are invisible to every threshold in a standard soak window. Catching them requires signals derived from business outcomes and at least one transaction performed by a human. This page sits under Post-Migration Monitoring.

When to Use This Approach

  • The site has a conversion path — purchase, signup, enquiry, booking — that generates revenue or leads.
  • The migration touched the application layer, the database, or any third-party integration.
  • Infrastructure monitoring reports health but something feels wrong.
  • Conversion reporting depends on analytics that the migration may itself have broken.
  • You need a trigger that fires on outcome rather than on availability.

Step-by-Step Instructions

1. Complete a Real Transaction by Hand

Before anything automated, have a person buy something, submit the form, or make the booking — and then verify the result at the far end: the order exists in the database, the email arrived, the payment appears in the provider’s dashboard. This single check exercises the entire chain in a way no assertion does, and it catches the majority of silent failures within minutes of cutover.

Do it within the first fifteen minutes, before any threshold has had time to accumulate a window. A hand-placed transaction is the fastest signal available about the conversion path and the only one that exercises every layer including the third parties, and the majority of silent failures are detectable this way while the rollback is still trivially cheap.

2. Compare the Outcome Ratio Against Last Week

Express conversions as a ratio against the same hour of the previous week rather than as a count or against the previous hour. Counts move with traffic, and an hour-on-hour comparison during a cutover scheduled in a trough reports a collapse that is entirely seasonal. The week-on-week ratio means the same thing at 03:00 as at midday.

-- Same hour, previous week: a ratio near 1.0 is healthy regardless of volume
SELECT COUNT(*) FILTER (WHERE ts >= NOW() - INTERVAL '1 hour')::float
     / NULLIF(COUNT(*) FILTER (WHERE ts BETWEEN NOW() - INTERVAL '8 days'
                                        AND NOW() - INTERVAL '8 days' + INTERVAL '1 hour'), 0)
FROM orders;

3. Measure at the Source, Not Through Analytics

Read the conversion count from the system of record — the orders table, the CRM, the payment provider — rather than from the analytics platform. A migration that drops a tracking snippet, changes a measurement identifier, or trips consent handling produces an instant and total collapse in reported conversions while the site works perfectly, and a threshold wired to that reporting will fire on a fault that does not exist.

Arrange access to the system of record before the window, not during it. This is the practical blocker in most organisations — the orders table belongs to another team, the payment dashboard needs an account nobody has requested — and a watcher who can see the error rate but not the conversion count is watching half the migration.

4. Instrument Each Step of the Funnel Separately

A single conversion number tells you something is wrong and nothing about where. Counting each step — product viewed, cart created, checkout started, payment submitted, order confirmed — localises the break to one transition immediately, which is usually enough to identify the cause without further investigation.

-- Step counts for the last hour; a cliff between two rows names the break
SELECT step, COUNT(*) FROM funnel_events
WHERE ts >= NOW() - INTERVAL '1 hour'
GROUP BY step ORDER BY MIN(step_order);

5. Probe the Path End to End on a Schedule

Where the conversion path can be exercised safely against a test account or a sandbox mode, run it as a synthetic transaction every few minutes and assert on the outcome rather than on the status code. A probe that follows the redirect and checks for 200 passes identically whether the order was created or silently discarded.

Silent failures and why each returns 200 Five conversion-path faults that produce no error signal, with what the visitor sees, what the logs record, and the signal that actually detects each. Every row logs a clean 200 and loses the transaction Fault Visitor sees Logs show Detected by Payment provider rejects origin IP confirmation page 200 funnel step count Form posts to a dead queue thank-you page 200 records at the far end Search index not migrated zero results 200 result-count metric Session store not shared logged out mid-flow 200 funnel step count Analytics tag dropped nothing wrong 200 source-of-record comparison The last row is the inverse case: the site is fine and the measurement is broken, which looks identical on any conversion dashboard.
None of these appear in an error rate, a latency percentile, or an availability check.

Worked Example

A retailer cuts over on a Thursday evening. Error rate stays at 0.2%, p95 latency improves, and the synthetic probe on /checkout returns 200 throughout. Everything looks better than before.

The funnel counts tell a different story by 22:00:

product_viewed     8412   (last week same hour: 8190)
cart_created       1104   (last week: 1071)
checkout_started    623   (last week: 610)
payment_submitted   601   (last week: 588)
order_confirmed       0   (last week: 571)

Every step is healthy until the last one. The payment provider is rejecting requests from the new origin’s IP range, which was not on their allow-list — a fault with no error signature on the site at all, because the application catches the rejection and renders a generic confirmation page anyway. Aggregate conversion would have shown a decline; the step breakdown named the exact transition in seconds.

Localising a break with per-step counts Funnel step counts compared against the same hour of the previous week, with every step healthy until one transition drops to zero, naming the failing step immediately. A cliff between two steps names the fault Product viewed healthy Checkout started healthy Payment submitted healthy Order confirmed zero An aggregate conversion number would show a decline and leave the cause entirely open. The step breakdown localises it to one transition in seconds, which usually identifies the responsible integration without further investigation.
Instrumenting the steps costs one event per transition and is the difference between knowing something is wrong and knowing what.

Record the step counts for the equivalent window before cutover so the comparison has something to sit against. Funnel shapes vary enough between sites that a general expectation is useless; what tells you a transition is broken is that it differs from the same transition last week on the same estate.

Verification

Confirm the detection works before relying on it.

# 1. A real transaction completes and appears at the far end
#    (place a live order, then confirm it in the orders table and the provider dashboard)

# 2. Funnel step counts are available for the last hour and the same hour last week
psql -c "SELECT step, COUNT(*) FROM funnel_events WHERE ts >= NOW() - INTERVAL '1 hour' GROUP BY step;"

# 3. The outcome ratio is computed from the system of record, not from analytics
psql -c "SELECT COUNT(*) FROM orders WHERE ts >= NOW() - INTERVAL '1 hour';"

A clean result is a hand-placed order visible in both the database and the payment provider, a full set of step counts for both windows, and a ratio sourced from the orders table rather than from a tag.

Where to read the conversion signal from Four possible sources for a conversion count — analytics, application logs, the system of record and the payment provider — ranked by how much the migration can distort each. Read from the source the migration is least able to break Source Distorted by the migration? Use as Analytics platform easily — tags, consent, ids secondary, and a tag check Application logs sometimes — shipping, format supporting detail System of record rarely the primary signal Payment provider almost never — external the arbiter A disagreement between the bottom two rows and the top one is itself a finding: the site works and the measurement does not.
Wire the rollback trigger to the third row, and use the fourth to settle any dispute about what really happened.

FAQ

Why not rely on the analytics platform for this? Because the migration is one of the most likely things to break it, and a broken tag and a broken checkout look identical in the report — both show conversions falling to zero. Reading from the system of record removes that ambiguity entirely: if the orders table is empty the business has a problem, and if the orders table is healthy while analytics reports nothing, the analytics has a problem. Watch both, and treat disagreement between them as informative rather than confusing.

How quickly should a conversion trigger fire? Slower than an error-rate trigger and faster than a daily report — thirty minutes is a reasonable window for most estates. Conversions are lumpier than requests, so a short window produces false alarms on ordinary variance; but an outcome fault is expensive per minute, so waiting for a daily figure is far too slow. Set the window from the volume: enough time to accumulate a statistically meaningful sample at your typical rate.

What if the site has no purchases to measure? Use the nearest equivalent that is recorded somewhere authoritative — form submissions landing in a CRM, signups in the user table, enquiries in a ticketing system. The requirement is not that money changes hands but that the outcome is written to a system you can count from. For a purely informational site, engaged sessions or internal search completions are serviceable proxies.

Does this need a separate on-call person from the infrastructure watch? No, but the watcher needs access to the outcome signal, which frequently belongs to a different team and a different tool. That access is the actual blocker in most organisations, and it needs arranging before the window rather than at 03:00 — a watcher who can see the error rate and cannot see the orders table is watching half the migration.

Should the synthetic transaction run against production? Where the payment provider offers a sandbox or test-card mode, yes — it exercises the real path with no financial effect. Where it does not, run it against production sparingly with a real low-value transaction and a scripted refund, or accept that the probe stops one step short of payment and rely on the funnel counts for that final transition. What does not work is probing a staging environment and inferring anything about production.

Related

← Back to Post-Migration Monitoring