Auditing Redirect Chains with a Site Crawler

Problem Statement

Map validation catches chains that exist within the map. It cannot catch the chains a migration creates by accident — a canonical-host rule that fires before the cross-domain rule, a trailing-slash normaliser at the edge, a CDN page rule nobody remembers, an application-level redirect that runs after all of them. Those only appear when something requests the URL and follows what comes back, which is what a crawler does. This page sits under Redirect Chain Elimination.

Where extra hops come from A single request passing through four layers — CDN page rule, host canonicalisation, trailing-slash normaliser and the redirect map — each of which can add its own hop. Four Layers, Four Chances to Add a Hop CDN page rule often undocumented +1 hop Host canonical apex to www +1 hop Slash normaliser adds or strips +1 hop Redirect map the intended hop the only one wanted Each layer is individually correct; the chain is a property of the stack, not of any one rule. No single team owns the result, which is why only a crawl finds it.
Chains are emergent — every layer behaves as designed and the visitor still takes four hops.

When to Use This Approach

  • The redirect map has passed its offline validation and you now need the served behaviour.
  • The stack has more than one layer that can redirect — a CDN plus an origin plus an application.
  • The Change of Address filing has been rejected and the map itself looks clean.
  • You are preparing a pre-launch sign-off and need evidence rather than spot checks.

Why Chains Survive Every Other Check

A chain is invisible to almost everything a team already does. Visitors do not notice it — two redirects complete in well under a second and the page that arrives is the right one. Uptime monitoring does not notice it, because every response in the sequence is a valid 301 and the final one is a 200. Application tests do not notice it, because they request the new URL directly. The redirect map’s own validation does not notice it, because within the map each rule is correct and singular.

The only observer that cares is one that requests the legacy URL and counts the responses. That is a narrow enough behaviour that it needs a deliberate tool, and it explains why chains are routinely discovered by a rejected Change of Address filing rather than by any of the checks that ran beforehand.

There is a second reason chains persist: no one team owns them. The CDN rule belongs to whoever administers the CDN account, the host canonicalisation to the infrastructure team, the trailing-slash behaviour to the framework’s defaults, and the redirect map to whoever ran the migration. Each will tell you, correctly, that their rule is right. The chain is a property of the composition, and composition has no owner unless somebody makes it their job.

# The one measurement nobody's existing tooling produces: responses per legacy URL
curl -sIL --max-redirs 10 https://legacy.example/product/xz-400 | grep -c '^HTTP'

That single number, taken across the whole legacy list, is the entire audit. Everything else in this page is about producing it reliably and attributing the excess.

Step-by-Step Instructions

1. Crawl the Legacy URL List, Not the New Site

Crawling the new site finds its internal links; it does not exercise the legacy URLs that carry the migration’s value. Feed the crawler the list mode input from your baseline export — the file produced by How to Export Full Crawl Data Before Migration.

2. Configure the Crawl So It Reports Hops

Three settings matter: follow redirects, report the full hop chain rather than only the final destination, and set the redirect limit high enough that a loop is recorded as a loop instead of an error. Screaming Frog’s list mode with “always follow redirects” produces the report; the same is available from any crawler that emits a redirect chain report.

3. Reproduce the Findings Without the Crawler

Whatever the crawler reports must be reproducible from a shell, because that is what you will hand to whoever owns the offending layer.

# Full hop chain for one URL: status and destination at each step
curl -sIL --max-redirs 10 https://legacy.example/product/xz-400 \
  | awk '/^HTTP/{s=$2} /^[Ll]ocation:/{printf "%s -> %s", s, $2} END{printf "%s final\n", s}'
# Hop counts across the whole legacy list, worst first
while read -r u; do
  printf '%s %s\n' "$(curl -sIL --max-redirs 10 "$u" | grep -c '^HTTP')" "$u"
done < legacy-urls.txt | sort -rn | head -30

4. Attribute Each Hop to a Layer

A chain report is only actionable once each hop has an owner. Compare the response headers at each step — the server, the CDN’s own headers and the cache status usually identify which layer answered.

# Which layer produced this hop? The response headers usually say.
curl -sI https://legacy.example/product/xz-400 | grep -iE '^(server|via|cf-ray|x-cache|location):'
Attributing hops to the layer that produced them A matrix mapping observable response header signatures to the responsible layer and the team that owns the fix. Reading the Headers to Find the Owner HEADER SIGNATURE LAYER WHERE THE FIX LIVES cf-ray, no via CDN edge rule provider dashboard server: nginx origin config repository app framework header application route application code no distinguishing header test from inside bypass the edge
When headers are ambiguous, request the origin directly with a Host header — if the hop disappears, the edge produced it.

5. Fix the Ordering, Then Re-crawl

Most chains are resolved by reordering rather than rewriting: make the cross-domain rule match before the host and slash normalisers, so the visitor arrives at the final URL in one step. Then re-crawl the same list and compare hop counts — the flattening techniques themselves are covered in Flattening Multi-Hop Redirect Chains into Single Hops.

Worked Example

A recruitment platform crawled 38,000 legacy URLs in list mode two days before its planned filing. The redirect report showed:

Hops URLs Attribution
1 9,400 correct
2 26,100 CDN “force www” page rule firing first
3 2,300 as above, plus a trailing-slash normaliser
loop 200 a locale rule matching its own output

The 200 looping URLs were the emergency; the 28,400 two- and three-hop URLs were the migration risk. Nobody had known the CDN page rule existed — it had been added years earlier by a marketing contractor and had been harmless until a second redirect layer was introduced.

Deleting the CDN rule and handling host canonicalisation once, at the origin, collapsed almost everything to a single hop. The re-crawl the next morning showed 37,800 URLs at one hop and 200 still looping, which were fixed by anchoring the locale pattern. The filing was accepted on the first attempt.

Verification

# 1. Distribution of hop counts across the full legacy list — the headline number
while read -r u; do curl -sIL --max-redirs 10 "$u" | grep -c '^HTTP'; done < legacy-urls.txt \
  | sort | uniq -c | sort -k2n

# 2. No URL exceeds two responses (one redirect plus the destination)
while read -r u; do
  n=$(curl -sIL --max-redirs 10 "$u" | grep -c '^HTTP')
  [ "$n" -le 2 ] || printf '%s hops: %s\n' "$n" "$u"
done < legacy-urls.txt | head -40

# 3. Both entry hosts behave identically — the check that catches canonical-host chains
for h in legacy.example www.legacy.example; do
  printf '%-24s %s\n' "$h" "$(curl -sIL --max-redirs 10 "https://$h/product/xz-400" | grep -c '^HTTP')"
done

Run the crawl again a week after launch. Chains reappear as new rules are added, and a single scheduled crawl comparing hop-count distribution to the previous run turns that into a report instead of a discovery — the same recurring-crawl pattern as Scheduling Recurring Crawls to Detect Pre-Migration Drift.

Hop-count distribution before and after the fix Paired bars for one, two, three hops and loops, showing the distribution collapsing to a single hop after reordering the rules. Hop Distribution Before and After 1 hop 2 hops 3 hops loops before after One reordering moved 28,400 URLs from two or three hops to one.
The distribution, not the total count, is the number worth reporting to stakeholders.

FAQ

Why does the crawler find chains the map validation did not? Because the map is one of several layers that can redirect. Validation proves the map is internally consistent; only a request through the real stack shows what the CDN, the host canonicaliser and the application add on top.

How many hops are acceptable? One. A single redirect from the legacy URL to the final destination is the target, and anything beyond that is latency for visitors and dilution for crawlers. Two hops is survivable in the short term; three or more should block a launch.

Should I crawl in list mode or spider mode? List mode, seeded with the legacy URL export. Spider mode follows links on the new site and will never request the legacy URLs that carry the migration’s history.

The crawler reports a loop but the page loads in my browser. Why? Usually a cookie or a session redirect that the browser satisfies on the second request and a stateless crawler never does — a locale or currency selector is the classic case. It is still a real defect: the crawler’s experience is what determines whether the URL can be indexed.

How often should the crawl be repeated after launch? Weekly for the first month, then monthly. Chains reappear whenever a rule is added at any layer, and comparing hop-count distribution to the previous run makes a regression visible in a single number.

Do I need a commercial crawler, or is curl enough? For a few thousand URLs, a shell loop over curl -sIL is genuinely sufficient and produces a report you fully understand. A dedicated crawler earns its place above roughly ten thousand URLs, where concurrency, resume and export matter, and where you also want the rendered-page checks it performs alongside the redirect report.

Should the crawl request the apex or the www host? Both, and separately. Legacy inbound links point at whichever host was canonical at the time they were created, which for an old site is frequently the one you no longer think about. A chain that only exists from the apex is still a chain for everybody arriving that way.

Does the crawler need to render JavaScript for this? Not for redirect chains — those are resolved from response headers before any rendering happens. Rendering matters for the separate question of whether the destination page has its content, which is worth checking in the same pass but is a different finding.

What about redirects issued by the application after authentication? They will show up as extra hops for a stateless crawler and may be legitimate for a signed-in user. Exclude authenticated paths from the audit, or crawl them with a session, but do not dismiss the finding without checking — a login redirect on a public URL is itself a migration defect.

How do I present this to stakeholders who do not care about hops? As a distribution and a trend, not as a total. “94% of legacy URLs now reach their destination in one step, up from 25% last week” is a sentence anybody can act on, and it maps directly to whether the filing will be accepted.

Related

← Back to Redirect Chain Elimination