Auditing Internal Link Graphs Before and After Migration

Problem Statement

Internal links are content, and they are the parity dimension that disappears most quietly. A template that drops its breadcrumb, its related-links block, or its section navigation removes thousands of internal links across the estate while changing nothing that looks wrong on any individual page — every page still renders, still returns 200, and still reads correctly. What changes is how authority and crawl attention flow, and the pages that suffer are the ones that were being reached primarily through the block that vanished. This page sits under Content Parity Auditing and covers comparing the link graph rather than the pages.

When to Use This Approach

  • Templates are being rebuilt, even where the content within them is unchanged.
  • Navigation, breadcrumbs, or related-content blocks are being redesigned.
  • A section is being consolidated, retired, or restructured.
  • Deep pages rely on category or hub pages to be discovered at all.
  • You need to know why a section stopped ranking when nothing about its pages changed.

Step-by-Step Instructions

1. Build the Graph From Both Crawls

Extract every internal link as an edge — source URL, target URL — from the frozen baseline and from a matching crawl of the new estate. The unit of analysis is the edge rather than the page, and both graphs must be built the same way for the comparison to mean anything.

# Edges from a crawl export: one source,target pair per line
awk -F, 'NR>1 {print $1 "," $4}' crawl-links.csv | sort -u > edges-legacy.csv
wc -l edges-legacy.csv edges-new.csv

Resolve every edge through the redirect mapping before comparing. On a migration that changes URLs, an unresolved comparison reports that essentially every edge is new and every old edge has vanished — technically accurate and completely useless. Rewriting the legacy targets to their post-migration equivalents is what makes the two graphs comparable at all.

2. Compare Inbound Counts, Not Outbound

Outbound links per page tell you what a template emits; inbound links per URL tell you how reachable a page is, which is the quantity that governs authority flow and crawl priority. A dropped navigation block shows as a small outbound change spread across a few templates and a large inbound change concentrated on the pages that block used to point at.

# Inbound degree per URL, before and after
cut -d, -f2 edges-legacy.csv | sort | uniq -c | awk '{print $2 "," $1}' | sort > in-legacy.csv
cut -d, -f2 edges-new.csv    | sort | uniq -c | awk '{print $2 "," $1}' | sort > in-new.csv

3. Find the Pages That Lost Reachability

Join the two inbound tables and rank by proportional loss. Absolute loss favours pages that were already heavily linked; proportional loss surfaces the page that went from four inbound links to one, which is the page most likely to fall out of the index entirely.

# Pages whose inbound link count more than halved
join -t, in-legacy.csv in-new.csv \
  | awk -F, '($3 - $2) / $2 < -0.5 {printf "%s  %d -> %d\n", $1, $2, $3}' \
  | sort -k2 -rn | head -30

Rank by proportional rather than absolute change. A category page dropping from 800 inbound links to 700 has lost more edges than an archive page dropping from three to one, and the archive page is the one at risk of leaving the index entirely. Sorting by proportion surfaces the pages whose remaining routes in are thin, which is where the real exposure sits.

4. Attribute the Loss to a Source Template

Take the lost edges and group them by the template of the source page. A loss spread evenly across many source templates is usually a content change; a loss concentrated in one source template is a block that was removed from that template, which is one defect rather than thousands.

# Which source templates stopped emitting links?
comm -23 edges-legacy.csv edges-new.csv \
  | cut -d, -f1 | sort | uniq -c | sort -rn | head -10

5. Check for Orphans and Newly Deep Pages

A page with zero inbound internal links is an orphan and will be found only through the sitemap, if at all. A page that moved from three clicks off the home page to six has not been orphaned but has been substantially deprioritised. Both are findings, and neither shows up in any per-page check.

# URLs present in the inventory but absent from the new edge set
cut -d, -f1 inventory-new.csv | sort -u > all-urls.txt
cut -d, -f2 edges-new.csv | sort -u > linked-urls.txt
comm -23 all-urls.txt linked-urls.txt | head -20
Why inbound degree is the measure that matters Outbound and inbound link counts compared by what each reveals, how a removed navigation block appears in each, and which pages the change actually affects. A removed block is small in one measure and huge in the other Measure A dropped block looks like Reveals Outbound per page a small drop on a few templates what changed Inbound per URL a large drop on thousands of pages who is affected Total edge count a number that fell very little on its own Orphan count pages appearing from nowhere the worst-affected pages Reporting only the total edge count is the common mistake: it registers the change and identifies nothing.
Inbound degree and orphan count together name the specific pages that lost their route in.

Worked Example

A retailer rebuilds its product template. The new template is cleaner, faster, and drops the “related products” block, which the design review recorded as a deliberate simplification. Nobody connects that decision to link structure.

The inbound comparison shows the consequence three weeks before launch:

$ join -t, in-legacy.csv in-new.csv | awk -F, '($3-$2)/$2 < -0.5' | wc -l
9418
$ comm -23 edges-legacy.csv edges-new.csv | cut -d, -f1 | sort | uniq -c | sort -rn | head -1
  47090 template:product

Forty-seven thousand edges came from one template, and 9,418 product pages lost more than half their inbound links. The worst-affected are older products that were reached almost entirely through the related-products block on their newer siblings — pages with no category placement and no external links, which after the migration would have had a single inbound link each from a paginated listing.

The team reinstates a reduced related-products block with six links instead of twelve, which recovers most of the graph at a fraction of the visual weight, and re-runs the comparison to confirm.

From edge lists to an attributed finding Edges extracted from both crawls, resolved through the redirect mapping, aggregated into inbound counts, and the lost edges grouped by source template to produce one finding per template. Resolve, count, difference, attribute Extract edges both estates Resolve via mapping old URLs to new Inbound per URL before and after Group lost edges by source template Skipping the resolution step makes every edge appear to have changed, because the target URLs moved with the migration. The final grouping is what turns 47,000 lost edges into one sentence: this template stopped emitting this block.
The same edge set also feeds the chain audit, so building it once serves two checks.

Record the pre-migration graph as a stored artefact alongside the crawl baseline. It cannot be rebuilt once the legacy estate stops serving, and the questions it answers — was this page always this isolated, or did we do that? — arrive weeks after the opportunity to capture it has passed.

Verification

Confirm the graph comparison is sound and the findings have been addressed.

# 1. Both graphs were built from comparable crawls
wc -l edges-legacy.csv edges-new.csv

# 2. No indexable URL has zero inbound internal links
comm -23 <(cut -d, -f1 inventory-new.csv | sort -u) <(cut -d, -f2 edges-new.csv | sort -u) | wc -l

# 3. No source template lost a disproportionate share of its outbound edges
comm -23 edges-legacy.csv edges-new.csv | cut -d, -f1 | sort | uniq -c | sort -rn | head -5

A clean result is comparable edge totals, zero orphaned indexable URLs, and no single source template accounting for an outsized share of the lost edges.

Severity of an inbound-link loss by the page it hits Four categories of page ranked by how much they depend on internal links for discovery, showing which losses are recoverable and which effectively remove a page from the index. The same edge loss is trivial or fatal depending on the page Page type Other routes in Effect of losing links Home and top categories external links, direct traffic negligible Popular articles external links, search modest Deep catalogue pages almost none may leave the index Archive and older content none at all effectively orphaned The bottom two rows are the pages nobody checks manually and the ones that depend entirely on the block that was removed.
Rank findings by the dependence of the affected pages, not by how many edges were lost.

FAQ

Should the comparison count nofollow links? Count them separately rather than excluding them. A link that was followed before and is nofollowed now represents a real change in how authority flows even though the edge still exists, and folding the two together hides it. Keep the attribute as a column on the edge and report both the total edge count and the followed subset.

Does link position within the page matter? It does, though rather less than link presence, and it is harder to measure reliably. A link in the main content is generally weighted more than one in a footer, so a template that moves its related-links block into the footer has degraded the graph without changing any count. Where the projection can capture the containing region cheaply, record it; where it cannot, prioritise presence and treat position as a secondary consideration.

What about links that changed target because of the migration? Resolve every edge through the redirect mapping before comparing, or the two graphs will appear to share almost no edges at all. This is essential rather than optional on a migration that changes URLs: without it the comparison reports that every link changed, which is technically true and completely useless.

How does this relate to the redirect chain audit? They are complementary and operate on the same edge set. Chain elimination asks whether an internal link points at a URL that redirects — which wastes a hop on every crawl — while this audit asks whether the link exists at all. Running both over the same graph is efficient, and a link that both redirects and only just survived a template change is worth fixing twice over.

Is a drop in total edges always bad? No. Removing genuinely low-value links — a tag cloud, an over-linked footer, duplicate navigation — improves the graph by concentrating authority where it matters. What makes a drop a defect is that it was unintended and that it concentrated on pages which had few other routes in. That is exactly why the attribution step matters: it separates a deliberate simplification from an accidental isolation.

Related

← Back to Content Parity Auditing