Restoring Redirect Rules from Version Control

Problem Statement

The redirect rules in the repository and the rules the server is actually applying are not the same file. Somebody added a rule through a control panel, an emergency fix was made by editing the config on the box, or a CDN rule was created in a web interface that has no repository at all. When you need to restore a known-good state you cannot simply deploy HEAD — that would silently discard the undocumented fixes that are currently holding the migration together. This page sits under Redirect Rollback & Recovery.

Three sources of truth that drift apart Repository rules, rules edited on the server, and rules created in a CDN interface all contribute to the served behaviour, and only the first is in version control. What Is Served Is Not What Is Committed Repository reviewed, in git Edited on the box no history at all CDN interface rules separate system Served behaviour the only ground truth Restore from git without reconciling the other two and you delete fixes nobody documented. Reconstruct the served state first; treat it as the baseline to diff against.
The served behaviour is the union of three sources, only one of which has a history you can read.

When to Use This Approach

  • You need to restore a previous redirect state and suspect the repository is not a complete record.
  • Emergency edits were made directly on a server or in a CDN dashboard during a cutover.
  • Ownership of the rules is split across teams — infrastructure owns the server config, marketing owns the CDN rules.
  • You are preparing to hand the migration over and want one authoritative, reviewable rule set.

How the Divergence Accumulates

No individual step in this drift is unreasonable. A cutover produces a fault at 03:00 and somebody fixes it on the box, because that is faster than a pipeline run and the alternative is leaving traffic broken. A marketing team adds a campaign redirect through the CDN dashboard, because that is the interface they have and the change is genuinely theirs to make. An infrastructure engineer adds a rate-limit rule that happens to live in the same file. Each is defensible; together they mean the repository no longer describes what is served.

What turns drift into an incident is the assumption that it has not happened. A restore performed on that assumption is not a restore — it is a new, unreviewed change that silently deletes an unknown set of fixes. Because the deletions are silent, the resulting breakage is attributed to whatever else was happening that day, and the same cycle repeats.

The reconciliation in this page is therefore not bookkeeping tidiness. It is the step that converts a dangerous operation into a safe one, and it is the reason the restore takes two hours rather than ten minutes. That ratio is uncomfortable during an incident, which is exactly why the capture in step one comes first: with the served behaviour recorded, you can proceed carefully and still prove afterwards that nothing was lost.

# The single most useful artefact in this whole process, captured in one line
curl -s -o /dev/null -w '%{http_code} %{redirect_url}\n' -I https://example.com/p/xz-400

Step-by-Step Instructions

1. Capture the Served State Before Changing Anything

The behaviour, not the file, is what you must not lose. Probe a representative sample of URLs and record what each currently does; this becomes the baseline you diff every candidate restore against.

# Snapshot current behaviour for a sample: status and destination per URL
while read -r u; do
  read -r code loc < <(curl -s -o /dev/null -w '%{http_code} %{redirect_url}' -I "$u")
  printf '%s\t%s\t%s\n' "$u" "$code" "$loc"
done < sample-urls.txt > served-before.tsv
wc -l served-before.tsv

2. Pull the Rules Out of Every System That Holds Them

# Server config as deployed, straight off the running host
ssh edge-01 'nginx -T' > /tmp/nginx-live.conf

# CDN rules exported through the API rather than read off a dashboard
curl -s -H "Authorization: Bearer $CF_TOKEN" \
  "https://api.cloudflare.com/client/v4/zones/$ZONE/rulesets" | jq '.result' > /tmp/cdn-rules.json

3. Diff the Live Config Against the Repository

# Normalise whitespace and comments so the diff shows real divergence only
normalise() { sed 's/#.*//; s/[[:space:]]\+/ /g; /^ *$/d' "$1" | sort; }
diff <(normalise /tmp/nginx-live.conf) <(normalise conf/nginx.conf) | head -40

Every line the live config has and the repository does not is an undocumented fix. Each one needs a decision — adopt it into the repository, or discard it deliberately — before any restore runs.

Deciding what to do with each divergent rule A matrix of four divergence types — emergency fix, panel-added rule, stale rule and drifted comment — with whether to adopt, discard or ignore each. Adopt, Discard, or Ignore DIVERGENCE STILL NEEDED? DECISION Emergency hotfix usually yes commit it, then restore Panel-added rule ask the owner move into the repo Rule for a dead path no discard deliberately Comment drift only n/a ignore
The point of the exercise is that each divergence gets a recorded decision rather than being resolved by whichever file is deployed last.

4. Rebuild a Restore Candidate, Then Deploy It

Construct the target state as a new commit — the known-good rules plus the adopted fixes — rather than checking out an old commit directly. A restore that is itself a commit is reviewable, and the next incident can diff against it.

# Build the candidate on a branch so the restore is reviewable, not a detached checkout
git switch -c restore/redirects-$(date -u +%Y%m%dT%H%M)
git checkout "$LAST_GOOD_SHA" -- conf/redirects/
git apply hotfixes-adopted.patch
git commit -m "Restore redirect rules to known-good state plus adopted hotfixes"

5. Compare Served Behaviour Before and After

# Re-probe the same sample and diff behaviour, not configuration
while read -r u; do
  read -r code loc < <(curl -s -o /dev/null -w '%{http_code} %{redirect_url}' -I "$u")
  printf '%s\t%s\t%s\n' "$u" "$code" "$loc"
done < sample-urls.txt > served-after.tsv
diff served-before.tsv served-after.tsv

Every line in that diff should be a change you intended. Anything else is collateral, and it is far cheaper to find it here than in the log triage described in Log-Based 404 Triage After a Cutover.

Worked Example

A travel operator ran a two-month migration during which four people made direct edits to the edge configuration. When a bad deploy required a restore, nginx -T on the live host differed from the repository by 37 lines.

Triaged, those 37 lines were:

  • 9 lines — a hotfix redirecting a supplier’s deep links that had been missed in the original map. Genuinely needed; committed before the restore.
  • 6 lines — rules pointing at a staging host, added during testing and never removed. Discarded.
  • 14 lines — a duplicate of rules that already existed in the repository under different formatting. Ignored.
  • 8 lines — a rate-limit block unrelated to redirects, added by the infrastructure team. Adopted into a separate file with its owner named in the commit.

The restore itself took ten minutes. The reconciliation took two hours and was the entire value of the exercise: without it, the nine-line hotfix would have vanished and a supplier’s traffic would have started 404ing with nobody connecting the two events.

Verification

# 1. The deployed config matches the branch you built — no drift reintroduced
ssh edge-01 'nginx -T' | sed 's/#.*//; s/[[:space:]]\+/ /g; /^ *$/d' | sort > /tmp/live.norm
sed 's/#.*//; s/[[:space:]]\+/ /g; /^ *$/d' conf/nginx.conf | sort > /tmp/repo.norm
diff /tmp/live.norm /tmp/repo.norm && echo "config in sync"

# 2. Behaviour diff contains only intended changes
diff served-before.tsv served-after.tsv | grep '^[<>]' | wc -l

# 3. CDN rules match their exported baseline
curl -s -H "Authorization: Bearer $CF_TOKEN" \
  "https://api.cloudflare.com/client/v4/zones/$ZONE/rulesets" | jq -S '.result' \
  | diff - cdn-rules-baseline.json && echo "cdn in sync"

Once all three pass, commit the exported CDN rules alongside the server config. A rule set that lives in two systems needs one export in the repository, or the same divergence starts accumulating the day after the incident is closed.

Restore as a forward commit rather than a checkout A branch line showing the known-good commit, the bad deploy, and a new restore commit that carries the good rules forward together with adopted hotfixes. Restore Forward, Never Backward known good the rules you want bad deploy stays in history restore commit good rules + hotfixes The restore is reviewable, attributable, and something the next incident can diff against. A detached checkout leaves no record of what was restored or why.
Treating the restore as a normal, reviewed change is what keeps the repository authoritative afterwards.

FAQ

Why not just deploy the last known-good commit? Because it discards every fix made since, and during a migration those fixes are often the ones holding traffic together. Reconcile first, adopt what is still needed, then restore.

How do I capture rules that only exist in a CDN dashboard? Export them through the provider’s API and commit the export. A dashboard rule with no exported representation is invisible to every review and every restore, and it will be the rule that surprises you.

Should the restore be a revert commit or a checkout of the old files? A new commit on a branch. It goes through review like any other change, and it leaves a record that says what was restored and which hotfixes were carried forward.

What is the fastest safe check that a restore worked? Behaviour, not configuration: re-probe the same sample of URLs you captured beforehand and diff the results. Configuration equality is a weaker signal, because the same behaviour can be produced by several different rule sets.

How do I stop divergence reappearing? Make the served config reproducible from the repository and check it periodically — a scheduled job that diffs nginx -T against the committed file and opens an issue on any difference. Divergence is not a one-time cleanup; without a check it re-accumulates within weeks.

Should emergency edits on the box be forbidden? No — forbidding them means the fault stays live while somebody waits for a pipeline. What should be required is that the edit is committed the same day, by the person who made it, with the reason in the message. The problem is never the emergency edit; it is the emergency edit that is never written down.

How do I diff a CDN rule set that has no text representation? Export it through the API to JSON, sort the keys, and commit that file. It is not the source of truth — the provider is — but a committed export makes divergence detectable, which is all the diff needs to do.

What if two systems disagree about the same path? The one closest to the user wins, which is usually the CDN. That precedence is worth writing down next to the rules, because during an incident people reason about the origin config and cannot understand why their change has no effect.

Is nginx -T safe to run on a production host? Yes — it validates and dumps the effective configuration without applying anything or interrupting traffic. It is the right way to answer “what is this server actually configured to do”, as opposed to reading the file you believe it loaded.

Should the reconciliation happen during the incident or afterwards? During, if the restore itself depends on it — which it does whenever you cannot enumerate what has changed. Deferring the reconciliation means performing the restore blind, and the whole point of the exercise is that a blind restore deletes fixes nobody has documented.

Related

← Back to Redirect Rollback & Recovery