Diffing Rendered DOM Between Legacy and New Templates

Problem Statement

Comparing raw HTML between two versions of a site produces a diff dominated by things nobody cares about — class names, attribute order, whitespace, a wrapper div that moved — while missing the thing that matters most, which is content that only exists after JavaScript has run. On a client-rendered template the source comparison reports both sides as nearly empty and concludes they match. What you want instead is a comparison of what a browser ends up displaying: the text, the headings, the links, the structured data present in the final DOM. This page sits under Content Parity Auditing and covers building that comparison so it produces a signal about substance rather than about markup.

When to Use This Approach

  • The new estate uses a different templating system, framework, or rendering strategy.
  • Any indexable content is rendered or hydrated client-side.
  • A source-level diff has produced thousands of differences and no findings.
  • You need per-template evidence rather than a per-page list.
  • The migration includes a redesign and you must separate intended changes from accidental ones.

Step-by-Step Instructions

1. Render Both Sides With the Same Engine and Settings

The comparison is only meaningful if the two captures differ in the site rather than in the measurement. Pin the browser version, the viewport, and the wait strategy, and use the same values for the legacy capture and the new one. A legacy capture taken with a three-second wait and a new capture taken on networkidle will differ for reasons that have nothing to do with either template.

// Capture the rendered text of the main content region, deterministically
const page = await browser.newPage();
await page.setViewport({ width: 1280, height: 900 });
await page.goto(url, { waitUntil: 'networkidle0', timeout: 30000 });
const text = await page.$eval('main', el => el.innerText.replace(/\s+/g, ' ').trim());

Capture the legacy side before the freeze and store it, exactly as with the crawl baseline. Rendered projections of the old estate cannot be recreated once it stops serving, and a comparison against a legacy capture taken after the migration started is a comparison against a moving target. Commit the projections alongside the baseline export so they share its provenance.

2. Extract Structure, Not Markup

Compare a normalised projection of the page rather than its HTML: the text content of the main region, the ordered list of heading levels and their text, the set of internal link targets, and the structured-data types present. Each of those is stable across a re-skin and sensitive to a content loss, which is exactly the sensitivity profile you want.

// A projection that survives a redesign and detects a content regression
const shape = await page.evaluate(() => ({
  words:    document.querySelector('main').innerText.trim().split(/\s+/).length,
  headings: [...document.querySelectorAll('main h1,main h2,main h3')].map(h => h.tagName + ':' + h.innerText.trim()),
  links:    [...document.querySelectorAll('main a[href^="/"]')].map(a => a.getAttribute('href')).sort(),
  schema:   [...document.querySelectorAll('script[type="application/ld+json"]')]
              .flatMap(s => { try { return [JSON.parse(s.textContent)['@type']].flat(); } catch { return ['INVALID']; } }),
}));

3. Sample by Template, Not by URL

Rendering is expensive — seconds per page rather than milliseconds — so a full-estate rendered crawl is impractical on anything large. Take a stratified sample instead: several pages per template class, chosen to include both a typical page and an unusual one, and treat the template rather than the page as the unit of finding.

# Three URLs per template class, favouring high-value pages
awk -F, 'NR>1 {c[$2]++; if (c[$2]<=3) print $1}' inventory-with-template.csv > render-sample.txt
wc -l render-sample.txt

Choose the sample deliberately rather than randomly. Three pages picked at random from a template class will usually be three typical pages, which agree with each other and reveal nothing about the range the template must handle. Pick a short page, a long one, and one with an awkward feature — an embedded table, a gallery, a long comment thread — because those are where rendering strategies diverge.

4. Diff the Projections and Report by Template

Compare the two projections field by field and aggregate the results per template. A word-count delta beyond tolerance, a heading that disappeared, a set of internal links that shrank, or a structured-data type that vanished are each a distinct finding — and reporting them grouped by template turns thousands of page-level differences into a handful of actionable defects.

# Report templates whose mean rendered word count dropped beyond tolerance
join -t, <(sort legacy-shapes.csv) <(sort new-shapes.csv) \
  | awk -F, '{d[$2] += ($8 - $3) / ($3 + 1); n[$2]++}
             END {for (t in d) if (d[t]/n[t] < -0.15) printf "%s %.0f%%\n", t, 100*d[t]/n[t]}'

5. Separate Intended Changes From Regressions

A redesign will legitimately change headings, remove blocks, and restructure content. Maintain an explicit list of what each redesigned template is supposed to lose or gain, and filter the report against it, so the output contains only differences nobody signed off. Without that filter a redesign produces a report consisting entirely of expected changes, which nobody will read twice.

Source comparison versus rendered comparison The two comparison approaches contrasted by what each is sensitive to, what each misses, and the noise level of the report each produces. One is loud and blind; the other is quiet and sighted Property Source diff Rendered diff Sensitive to markup changes extremely barely Sees hydrated content never yes Survives a re-skin no — every page differs yes Cost per page milliseconds seconds Signal-to-noise very low high The cost row is the only one favouring source comparison, and it is what makes stratified sampling necessary rather than optional.
Use source comparison for cheap full-estate sweeps of status and metadata; use rendered comparison for substance.

Worked Example

A publisher moves from a server-rendered template to a framework that hydrates article bodies client-side. The source-level diff looks reassuring: both versions emit a similar amount of HTML and the structural markup is comparable.

The rendered comparison tells a different story on one template. Articles with more than about 1,200 words render only their first section, because the new template lazily loads the remainder on scroll and the crawler never scrolls:

template=article  legacy_words=1840  new_words=612  delta=-67%
template=article  legacy_words=940   new_words=940  delta=0%
template=category legacy_words=310   new_words=318  delta=+3%

Short articles are unaffected, which is why manual spot-checking missed it — the pages people opened happened to be short. The fix is to render the full body server-side and enhance on the client rather than the reverse. A re-run of the same comparison confirms the delta returns to within tolerance across the whole sample.

What to extract into the projection Four projected fields — word count, heading outline, internal link set and structured-data types — with what each detects and why it survives a redesign. Project four things, and a re-skin moves none of them Field Detects Changed by a re-skin? Rendered word count truncated or unrendered content no Heading outline a section dropped from the template rarely Internal link set a navigation block removed sometimes Structured-data types markup lost in the rebuild no Each field is chosen for the same property: high sensitivity to content loss, low sensitivity to presentation change.
Comparing element counts or class names instead would invert that and produce a report nobody can act on.

Assert on the projection fields individually rather than on an overall similarity score. A single number tells you two pages differ and nothing about how, whereas a per-field comparison says “headings match, links match, word count fell 40%” — which names the defect and frequently identifies the cause without further investigation.

Verification

Confirm the comparison itself is sound before trusting its findings.

# 1. The harness is deterministic: same URL twice, same projection
node render-shape.js https://legacy.example.com/article/x > a.json
node render-shape.js https://legacy.example.com/article/x > b.json
diff a.json b.json && echo "deterministic"

# 2. Every template class appears in the sample
awk -F, 'NR>1 {print $2}' inventory-with-template.csv | sort -u > all-templates.txt
awk -F, '{print $2}' render-sample-shapes.csv | sort -u | comm -23 all-templates.txt -

# 3. Rendered word counts exceed unrendered ones where hydration is expected
node render-shape.js --no-js https://new.example.com/article/x | jq .words
node render-shape.js         https://new.example.com/article/x | jq .words

A clean result is an identical projection across two runs of the same URL, no template missing from the sample, and a rendered word count that materially exceeds the unrendered one on templates known to hydrate.

Building a rendered parity comparison A stratified sample drawn per template, rendered with pinned settings on both estates, projected into comparable fields, then diffed and reported per template class. Sample, render, project, report by template Sample per template three varied pages Render both sides identical settings Project four stable fields Report per template not per page Pinning the browser version, viewport and wait strategy is what stops the harness itself becoming a source of differences. Reporting per template converts thousands of page-level diffs into a handful of defects with owners.
Run it against staging before the freeze and again against production after cutover — the second run is nearly free once the harness exists.

FAQ

Why not just compare the HTML source? Because it answers a different question. Source comparison is sensitive to every markup change and blind to every rendering-dependent one, which on a modern estate means it produces maximum noise on exactly the templates where it has minimum signal. Rendered comparison inverts both: a re-skin barely moves the projection, and content that fails to appear is impossible to miss.

How many pages per template are enough? Three is usually sufficient to detect a template-level regression, provided they are chosen for variety rather than convenience — a short page, a long one, and one with unusual features such as embedded media or a table. The failure mode to avoid is sampling three near-identical pages, which will agree with each other and tell you nothing about the range the template has to handle.

Does this replace visual regression testing? No — they answer different questions and both are worth having. Visual regression compares appearance and catches layout and styling problems; this compares extracted content and catches substance problems. A page can be pixel-identical while having lost its structured data, and can look entirely different while carrying exactly the same content.

What about content behind interaction, like tabs or accordions? Decide first whether it needs to be in the DOM at all. Content that only exists after a click is content a crawler will not see, which may be intended for genuinely supplementary material and is a serious problem for anything that should rank. Where it must be indexable, render it into the DOM and hide it with CSS rather than injecting it on interaction — and have the projection assert its presence so a later change cannot quietly remove it.

How long does a rendered crawl take? Seconds per page rather than milliseconds, which is precisely why the sampling in step 3 exists. A few hundred rendered pages is a comfortable overnight job and enough to cover every template class several times over; a full rendered crawl of a large estate is neither necessary nor practical for this purpose.

Related

← Back to Content Parity Auditing