Log-Based 404 Triage After a Cutover

Problem Statement

The 404 log is the fastest and most direct signal about redirect coverage after a cutover, and by default it is unusable. A healthy site already carries a steady background of requests for paths that never existed — vulnerability scans, malformed URLs, typo’d links from other sites — and a migration adds every URL you deliberately retired on top of that. The result is thousands of entries in which a genuine defect affecting real traffic is indistinguishable from noise. Triage is the process of subtracting what you expected, grouping what remains, and ranking it so the top of the list is the thing worth fixing. This page sits under Post-Migration Monitoring.

When to Use This Approach

  • A cutover has just completed and redirect coverage needs verifying against real traffic.
  • The 404 report contains too many entries to read.
  • You have an intentional-retirement list from the audit and want to use it.
  • A section is suspected of being unreachable but nobody can say which URLs.
  • You need evidence for the rollback decision rather than an impression.

Step-by-Step Instructions

1. Rank by Frequency Before Anything Else

Sort by request count descending and nothing else. A single mis-scoped redirect rule produces thousands of identical failures and sits unmissably at the top, while genuinely isolated problems trail off below — so the first pass costs one command and usually identifies the largest defect immediately.

# Frequency-ranked 404 paths from the new origin
awk '$9 == 404 {print $7}' access.log | sort | uniq -c | sort -rn | head -40

Resist the temptation to read the list before ranking it. A 404 report read top to bottom in log order is a stream of individually plausible entries that suggests hundreds of separate problems; the same data ranked by count usually shows one entry accounting for most of the volume. The ordering is what makes the first defect obvious rather than buried.

2. Subtract the Intentional Retirements

Every URL you deliberately retired during the mapping phase will appear here, and it is not a defect. Filtering the retirement list out is what converts an undifferentiated pile into a list of surprises — and it is the specific payoff for having recorded those decisions rather than letting retirements happen by omission.

# Exclude paths we meant to retire, then re-rank
awk '$9 == 404 {print $7}' access.log | sort | uniq -c | sort -rn \
  | grep -vFf intentional-410s.txt > unexpected-404s.txt
head -30 unexpected-404s.txt

3. Group by Pattern, Not by URL

Individual URLs are rarely the unit of the defect. Collapse paths to their shape — replacing identifiers, dates and slugs with placeholders — and the list usually reduces from hundreds of entries to a handful of patterns, each corresponding to one missing or mis-scoped rule.

# Normalise identifiers and dates so the pattern behind the URLs emerges
awk '{print $2}' unexpected-404s.txt \
  | sed -E 's#/[0-9]{4}/[0-9]{2}/#/YYYY/MM/#; s#/[0-9]+#/N#g; s#/[a-z0-9-]{20,}#/SLUG#g' \
  | sort | uniq -c | sort -rn | head -20

Choose the normalisation carefully enough to group without over-collapsing. Replacing every numeric segment with a placeholder will merge genuinely different sections if both use numeric identifiers, which hides one defect inside another. Normalise identifiers, dates and long slugs, and leave the structural path segments intact — the goal is to see the shape of the URL, not to reduce everything to a single line.

4. Separate Referred Traffic From Direct

A 404 reached from an internal link is a defect in your own estate and is entirely within your control; one reached from an external referrer is a redirect gap; one with no referrer at all is often a bookmark, a bot, or an old email. The remedy differs in each case, and the referrer column is what tells them apart.

# Internal referrers first — these are links you are still emitting
awk '$9 == 404 && $11 ~ /example\.com/ {print $7, $11}' access.log \
  | sort | uniq -c | sort -rn | head -20

5. Re-run on a Schedule, Not Once

Redirect gaps surface as crawlers and users work through the long-lived parts of the estate, which happens over days rather than in the first hour. A single pass on cutover night catches the obvious cases and misses the archive entirely — so re-run the triage every few hours through the soak window and daily for a fortnight after.

Classifying a 404 by where the request came from Four referrer categories for a post-migration 404 — internal link, external site, search engine and no referrer — with what each implies and the appropriate remedy. The referrer decides whose defect it is Referrer Means Remedy Your own site you are still linking to it fix the link, not the redirect An external site a genuine redirect gap add the mapping A search engine the URL is still indexed add the mapping, expect decay None bookmark, bot, or old email map if volume justifies it The first row is the one people mis-triage as a redirect problem when it is an internal linking problem with a different owner.
Sorting by referrer before by count separates defects you control from gaps in the mapping.

Worked Example

A publisher completes a cutover and finds 41,000 404s in the first six hours. Ranked by frequency, the top entry accounts for 9,200 of them:

   9218 /2019/03/some-article/
   4471 /2018/11/another-article/
   3902 /2020/07/a-third/

Subtracting the retirement list changes nothing — none of these were meant to be retired. Grouping by pattern collapses all three and several hundred more into one line:

  38104 /YYYY/MM/SLUG/

Every dated article URL is failing. The redirect rule for the archive was written against /blog/YYYY/MM/ while the legacy structure used a bare /YYYY/MM/, so the pattern matched nothing at all. One corrected rule resolves 93% of the log. The remaining 2,900 entries turn out to be a genuine mixture — scans, typos, and about forty real gaps worth mapping individually.

From 41,000 log lines to one broken rule A raw 404 log ranked by frequency, filtered against the intentional retirement list, collapsed to URL patterns, and resolved to a single mis-scoped redirect rule. Rank, subtract, group — usually one rule Raw log 41,000 entries Subtract retirements what we expected Collapse to patterns hundreds to a handful One rule 93% of the log Each step is one command and each reduces the list by an order of magnitude. The pattern-collapse step is the one that turns a list of URLs into a description of a defect somebody can fix.
What remains after the fix is the genuinely mixed tail — scans, typos, and the small number of real gaps worth mapping individually.

Feed the confirmed gaps back into the mapping inventory rather than adding rules directly to the server config. A rule added under pressure and never recorded is a rule the next regeneration silently drops, which returns the same 404s weeks later with nobody able to explain why they came back.

Verification

Confirm the triage is complete and the fixes actually landed.

# 1. Unexpected 404s as a share of total requests, after filtering
tot=$(wc -l < access.log)
unx=$(awk '$9 == 404 {print $7}' access.log | grep -vFf intentional-410s.txt | wc -l)
awk -v u="$unx" -v t="$tot" 'BEGIN {printf "unexpected 404 rate: %.2f%%\n", 100*u/t}'

# 2. No internal referrer is still producing a 404
awk '$9 == 404 && $11 ~ /example\.com/' access.log | wc -l

# 3. The previously top-ranked pattern no longer appears
grep -c '/2019/03/' access.log

A clean result is an unexpected 404 rate below 1%, zero 404s reached from internal links, and the previously dominant pattern absent from subsequent log windows.

Why the triage runs for weeks rather than hours How different parts of an estate surface in the 404 log over time, from immediately-requested popular pages to archive URLs that may not be re-crawled for weeks. The archive shows up long after cutover night Part of the estate Appears in the log Found by a one-off pass? Popular pages within minutes yes Category and hub pages within hours yes Long-tail content over days partly Deep archive over weeks no A single pass on cutover night covers the top two rows, which is also the part you were most likely to have mapped correctly.
Automate the report so the long tail costs nothing to keep watching.

FAQ

How does this interact with the CDN in front of the origin? It changes where you have to look. An edge that serves a cached 404, or that answers some paths without ever consulting the origin, means the origin log is an incomplete record of what visitors experienced — and on a heavily cached estate it can be substantially incomplete. Pull the edge logs as well and triage the union, or at minimum confirm which paths the edge answers independently so you know what the origin log cannot tell you. The same applies to any redirect issued at the edge: it never reaches the origin at all.

Should retired URLs return 404 or 410? 410 where the retirement is deliberate and permanent, because it states that the absence is intentional and crawlers de-index it faster than they do a 404. The operational benefit during a migration is just as valuable: a 410 is a positive signal in your own logs that a request matched a decision somebody made, which makes the triage above considerably cleaner than filtering by an external list.

How much 404 traffic is normal? Below about 1% of requests on a settled site, and higher for the first days after a cutover as crawlers work through the legacy estate. What matters is the trend and the composition rather than the absolute figure: a stable rate composed of scans and typos is fine, and a rising rate or one dominated by a single pattern is not.

What about 404s from bots and vulnerability scans? Filter them into a separate bucket and eyeball the top entries occasionally rather than discarding them silently. They are noise for triage purposes, but a surprising volume of requests to a nonsense path sometimes turns out to be a real integration calling a URL nobody documented — which is exactly the class of orphan the audit exists to find.

Can this replace a crawl-based redirect audit? No, and the two are complementary. The log tells you what is actually being requested, which the crawl cannot know; the crawl tells you about URLs that exist and are not yet being requested, which the log cannot know. Run the crawl before the cutover to build coverage and the log triage after it to find what the crawl missed.

How long should the triage keep running? Frequently through the soak window, daily for a fortnight, then weekly for a couple of months. The tail matters because deep archive URLs may not be re-crawled for weeks, and a gap affecting them will not appear in the first days at all. Automating the report is what makes the long tail affordable — it costs nothing to keep running once written.

Related

← Back to Post-Migration Monitoring