Validating Redirect Maps Before Deployment

Problem Statement

A redirect map is a data file that becomes production behaviour, and it is usually reviewed the way a spreadsheet is reviewed — by scrolling. Twenty thousand rows hide a handful of rows that will produce a loop, point at a URL that does not exist, or map the same source twice with different destinations. Every one of those is deterministically detectable before deployment, and every one of them is expensive to find afterwards from access logs. This page sits under CSV Mapping Workflows.

When to Use This Approach

• A redirect map is generated by a script or exported from a spreadsheet by a non-engineer.
• The map has more rows than anyone will read carefully — in practice, more than a few hundred.
• Multiple people contribute rows, so duplicate and conflicting sources are likely.
• You want a pipeline gate rather than a review convention.

A Data File That Becomes Behaviour

The reason redirect maps escape review is a category error. They arrive as spreadsheets, they are produced by people who do not think of themselves as shipping code, and they are reviewed the way data is reviewed — by looking at a sample and checking that it seems right. But the moment the file is deployed it is behaviour: every row is a conditional that runs on every matching request, and a wrong row is a production defect with the same blast radius as a wrong line of code.

Treated as code, the map’s defects are unremarkable. A self-redirect is an infinite loop. A cycle across rows is the same loop spread over several statements. A chain is an unnecessary indirection. A duplicate source is two conflicting definitions where only one can win, and which one wins depends on the server’s matching order rather than on anybody’s intent. Every one of these has an obvious automated check, and none of them requires a running server to detect.

What makes this worth building rather than doing by eye is scale asymmetry. The map has tens of thousands of rows; the defects number in the tens. A human reviewer’s attention is uniformly distributed and the defects are not, so reading the file is close to the worst possible use of the time. A script’s attention is perfectly distributed and costs seconds.

# The scale problem in one line: nobody is reviewing this by eye
wc -l map.csv && cut -d, -f1 map.csv | sort | uniq -d | wc -l

The rest of this page is the four checks worth automating, in the order that fails fastest.

Step-by-Step Instructions

1. Normalise Before You Compare

Most false positives and most missed defects come from comparing unnormalised strings. Decide once how you treat trailing slashes, case, the host, default ports and query ordering, then apply it to both columns.

# One normalisation applied to both columns — comparisons are meaningless without it
from urllib.parse import urlsplit, urlunsplit

def norm(u: str) -> str:
    s = urlsplit(u.strip())
    path = s.path.rstrip("/") or "/"          # trailing slash is not a distinction here
    return urlunsplit(("", "", path.lower(), s.query, ""))

2. Run the Structural Checks

# Loops, chains, duplicates and self-redirects, from the map alone
import csv, collections

rows = [(norm(r["source"]), norm(r["target"])) for r in csv.DictReader(open("map.csv"))]
sources = collections.Counter(s for s, _ in rows)
target_of = dict(rows)

self_redirects = [s for s, t in rows if s == t]
duplicates     = [s for s, n in sources.items() if n > 1]
chains         = [(s, t) for s, t in rows if t in sources]

def cycle_from(start):
    seen, cur = [], start
    while cur in target_of:
        if cur in seen:
            return seen[seen.index(cur):]
        seen.append(cur)
        cur = target_of[cur]
    return None

cycles = {tuple(c) for s in sources if (c := cycle_from(s))}

for name, items in [("self-redirect", self_redirects), ("duplicate source", duplicates),
                    ("chain", chains), ("cycle", cycles)]:
    print(f"{name:18s} {len(items)}")

3. Check the Destinations Exist

The structural checks need no network; this one does. Sample rather than fetching every row, and always fetch the whole set for high-value URLs.

# Concurrent HEAD requests over the destination column, reporting anything that is not 200
cut -d, -f2 map.csv | tail -n +2 | sort -u \
  | xargs -P 12 -I{} sh -c 'printf "%s %s\n" "$(curl -s -o /dev/null -w "%{http_code}" -I "{}")" "{}"' \
  | grep -v '^200 ' | head -40

4. Confirm No Rule Matches Its Own Output

Regex rules deserve a check of their own: apply each rule to its own destination and fail if it matches. This single assertion catches the most common production loop.

# A rule that rewrites its own output is the classic redirect loop
import re

def rule_matches_own_output(pattern, replacement, sample):
    dest = re.sub(pattern, replacement, sample)
    return bool(re.search(pattern, dest))

assert not rule_matches_own_output(r"^/product/(.*)$", r"/p/\1", "/product/xz-400")

5. Gate the Pipeline on It

Exit non-zero on any structural failure and on any destination that is not 200 within the high-value set. A validation suite that reports and continues is a report, not a gate.

Worked Example

A B2B catalogue generated its redirect map by joining a legacy URL export to a new-slug table in a spreadsheet. The file had 11,400 rows and passed visual review twice.

The validation suite found four classes of defect in under a minute:

The 1,208 chains were the significant finding. Every one would have worked for a visitor and taken two hops, and collectively they would have failed the Change of Address sampling described in Using the Change of Address Tool Correctly. Resolving each chain to its final destination — a five-line transitive closure over the map — reduced the file to 9,800 rows, all single-hop.

Verification

# 1. The suite fails loudly on a deliberately broken map — test the test
printf 'source,target\n/a,/a\n' > /tmp/bad.csv
python3 validate_map.py /tmp/bad.csv && echo "SUITE IS BROKEN" || echo "suite correctly failed"

# 2. Post-deploy: sample the live behaviour and confirm one hop per URL
shuf -n 200 sources.txt | while read -r u; do
  n=$(curl -sIL --max-redirs 5 "$u" | grep -c '^HTTP')
  [ "$n" -le 2 ] || echo "CHAIN: $u"
done

# 3. Row count in the deployed map matches the validated file
ssh edge-01 'wc -l < /etc/nginx/redirects/map.conf'
wc -l < build/map.conf

Check one deserves emphasis. A validation suite is code, and a suite that silently passes everything is worse than no suite because it produces confidence. Feed it a known-bad file in CI and assert that it fails.

FAQ

Should the intermediate URL be removed from the map after flattening? No — it may still be requested directly, so it needs its own row pointing at the final destination. What flattening removes is other rows pointing at it, which is what created the chain.

How large a sample is enough for the destination check? Fetch every destination in the high-value set and a random sample of a few hundred elsewhere. Full coverage on tens of thousands of rows is possible but slow enough that people start skipping the gate, which is worse than sampling.

Which normalisation decisions matter most? Trailing slash and case, in that order. Both routinely produce phantom differences that hide real duplicates, and both must be applied identically to the source and target columns or the comparisons mean nothing.

Can this run as a pre-commit hook rather than in CI? The structural stages can — they need no network and finish in seconds. Keep the destination checks in CI, where a slow or flaky network delays a build rather than blocking somebody’s commit.

What should happen when a destination returns 404? Fail the build and fix the row. A redirect to a 404 is worse than no redirect: it consumes the crawl, satisfies naive “does it redirect” checks, and leaves the visitor at a dead end with no signal that anything is wrong.

How should duplicate sources be resolved? By deciding, not by letting the server decide. Two rows with the same source mean somebody intended two different destinations, and whichever wins is an accident of matching order. Fail the build, get the owner to pick, and record the choice in the file.

Do regex rules need the same treatment as literal rows? More, not less. A literal row can only be wrong about one URL; a regex row can be wrong about a whole template, and it is the only rule type that can match its own output. The self-match assertion is cheap and catches the single most common production loop.

Should the validation run against the CSV or the generated server config? Both, ideally, but the CSV is where the fix belongs. Validating the generated config catches template bugs in the generator; validating the CSV catches the data problems, which are the great majority. If you only do one, do the CSV and diff the generated output separately.

What is the right response to thousands of chains? Compute the transitive closure and rewrite the file. Chains almost never need individual attention — they are the mechanical consequence of a map assembled in stages, and resolving each source to its final destination is a few lines of code that fixes all of them at once.

How do I keep the suite from becoming an obstacle? Split it by cost, as above, and make the fast stages genuinely fast. A gate that adds ten seconds to a build is invisible; one that adds ten minutes gets bypassed, and a bypassed gate provides no protection while still providing the feeling of it.

Related

• CSV Mapping Workflows
• Using Python to Generate CSV Redirect Maps
• Redirect Chain Elimination

← Back to CSV Mapping Workflows