Diagnosing Crawled — Currently Not Indexed After a Move

Problem Statement

Three weeks after a migration the coverage report shows tens of thousands of URLs under Crawled — currently not indexed. The pages return 200, the redirects are single-hop, the sitemap is clean, and nothing in the report explains why the crawler looked at a page and decided not to keep it. The state is genuinely ambiguous: it covers a normal reprocessing queue, a template that lost its content, and a duplication problem, and the three demand opposite responses. This page sits under Index Coverage Handoff.

Decision tree for a crawled-not-indexed spike A decision tree splitting the affected URL set by whether the rendered page has content, whether it duplicates another URL, and whether the count is falling week on week. Three Causes, Three Different Answers Affected URL set export and segment Rendered page is empty template or JS fault fix and request recrawl Duplicates another URL parameters or variants canonicalise, don't chase Count falling weekly reprocessing backlog wait, keep measuring Segment the set before acting — the three causes usually coexist in the same report.
The report gives one label to three unrelated situations, so the first move is always to split the URL list.

When to Use This Approach

  • The count under this state rose sharply after a cutover rather than drifting up over months.
  • Affected URLs return 200 and are present in the sitemap.
  • The redirect map has already been validated, so this is not a redirect fault wearing a different label.
  • You need to tell stakeholders which part of the number is a defect and which is a queue.

Selection, Not Discovery

The instinct when a large number of URLs are not indexed is to help the crawler find them: re-submit the sitemap, request indexing, add internal links. All of that addresses discovery, and this report state is explicit that discovery already happened. The crawler found the page, fetched it, rendered it, and decided not to keep it. The constraint is selection.

Selection is a judgement about value relative to everything else competing for the same index. A page that is thin, duplicated by another URL, or substantially similar to a hundred sibling pages loses that judgement regardless of how loudly it is advertised. This is why re-submission changes nothing: it re-asserts an answer to a question that was not asked.

What makes the state confusing after a migration specifically is that all three underlying causes spike at once. Reprocessing genuinely does queue — millions of URLs changed address in a short period, and working through them takes weeks. Templates genuinely do break, because they were rewritten. Duplication genuinely does appear, because new routing layers manufacture URL variants that never existed before. A number that combines all three cannot be interpreted without being split first.

# Before doing anything else: how many, and are they concentrated anywhere?
wc -l not-indexed.csv
cut -d/ -f4 not-indexed.csv | sort | uniq -c | sort -rn | head

The good news in this framing is that two of the three causes are fixed by removing pages rather than by adding work, and the third fixes itself. Almost no version of the correct response involves the effort teams usually expend here.

Step-by-Step Instructions

1. Export the URL List and Segment It

Do not work from the chart. Export the affected URLs, join them to your template map, and count per template — the same join used to produce the segmented sitemap in Splitting Large Sitemaps for Faster Recrawl.

# Group the exported URL list by template so the spike becomes attributable
import csv, re, collections

TEMPLATES = [("product", r"^/p/"), ("category", r"^/c/"),
             ("article", r"^/blog/"), ("archive", r"^/\d{4}/")]

counts = collections.Counter()
with open("not-indexed.csv") as fh:
    for row in csv.DictReader(fh):
        path = re.sub(r"^https?://[^/]+", "", row["url"])
        counts[next((n for n, p in TEMPLATES if re.match(p, path)), "other")] += 1

for name, n in counts.most_common():
    print(f"{n:8d}  {name}")

2. Check What the Page Actually Renders

A template that lost its main content during the migration looks fine to a browser with a warm cache and empty to a crawler. Fetch without JavaScript and count the words that survive.

# Words present in the raw HTML, before any client-side rendering
curl -s https://example.com/p/example-product \
  | sed 's/<script[^>]*>.*<\/script>//g; s/<[^>]*>/ /g' | wc -w

Fewer than a couple of hundred words on a page that used to be substantial is your answer. Compare against the legacy rendering using the method in Diffing Rendered DOM Between Legacy and New Templates.

3. Look for Accidental Duplication

Migrations create duplicates by accident: a new routing layer that serves the same page at /p/123 and /p/123/, a filter that generates a URL per combination, a locale prefix applied to pages that have no locale variants.

# Same content on two URLs? Compare normalised body hashes
for u in https://example.com/p/123 https://example.com/p/123/; do
  printf '%s  %s\n' "$(curl -s "$u" | tr -d ' \n\t' | md5sum | cut -c1-12)" "$u"
done

Identical hashes mean you have a canonical problem, not an indexing problem.

Distinguishing the three causes by observable signal A matrix comparing template fault, duplication and reprocessing backlog across rendered word count, duplicate hash, and week-on-week trend. Telling Them Apart Without Guessing CAUSE RENDERED WORDS DUPLICATE HASH WEEKLY TREND Template fault collapsed no flat or rising Duplication normal yes flat Reprocessing normal no falling Three cheap measurements settle a question that otherwise turns into a month of opinion.
Each row is separable with a single command, which is why this diagnosis should take an afternoon rather than a sprint.

4. Measure the Trend Before Escalating

Plot the count weekly. A backlog falls; a defect does not. Two consecutive weeks of decline on a template with healthy rendering and no duplicates is the signal to stop investigating and let the queue drain.

Worked Example

A recipe site moved to a new framework and a new domain in the same release. Six weeks later 31,000 of 44,000 URLs sat under crawled, currently not indexed, and the working assumption in the team was that the domain change had failed.

Segmenting by template split the number cleanly:

Segment Affected Rendered words Duplicate hash Trend
recipes 1,900 ~900 no falling
tag pages 26,400 ~40 yes, to each other flat
author pages 2,700 ~60 no flat

The recipes were a backlog and resolved themselves. The tag pages were a new framework’s automatic taxonomy routes — 26,400 near-identical pages that had never existed on the legacy site and should never have been in the sitemap. The author pages were a genuine template fault: the biography field was never mapped in the content export.

Two of the three findings were fixed by removing pages rather than by adding effort, and the migration itself turned out to have worked.

Verification

# 1. Rendered word count for a sample of the affected template
while read -r u; do
  n=$(curl -s "$u" | sed 's/<script[^>]*>.*<\/script>//g; s/<[^>]*>/ /g' | wc -w)
  printf '%6d  %s\n' "$n" "$u"
done < affected-sample.txt | sort -n | head

# 2. Canonical on the affected page points at itself, not elsewhere
curl -s https://example.com/p/123 | grep -o '<link rel="canonical"[^>]*>'

# 3. Week-on-week direction, taken from two exports a week apart
wc -l not-indexed-week1.csv not-indexed-week2.csv

If the canonical in check two points at another URL, the page is not meant to be indexed and the report is describing your own instruction back to you. That is a correct outcome, not a defect — but it should be reflected by removing the URL from the sitemap so the submitted count stops overstating what you asked for.

Expected decline curve after a migration A falling line showing the crawled-not-indexed count declining from week one to week ten, with a flat comparison line indicating a defect that will not resolve itself. A Backlog Falls, A Defect Does Not defect — flat backlog — falling week 1 week 5 week 10 Two flat weeks on a healthy template is the point at which waiting stops being a strategy.
The shape of the curve is the diagnosis; the absolute number on any single day is not.

FAQ

Is this state an error? No, it is a decision. The crawler fetched the page and chose not to index it, usually because the page is thin, duplicated, or lower value than others competing for the same budget. Treating it as an error leads teams to re-submit sitemaps repeatedly, which changes nothing.

How long should I wait before treating it as a defect? Two consecutive weeks with no decline, on a template whose rendered content is intact and whose canonical points at itself. Before that, you are most likely watching a reprocessing queue that will drain on its own.

Does requesting indexing help? For a handful of important URLs after a genuine fix, yes — it queues them explicitly. As a response to a spike of tens of thousands it does nothing useful, because the constraint is selection rather than discovery.

Can redirects cause this state? Indirectly. If the destination of a redirect is thin or duplicated, the redirect worked and the destination still will not be kept. That is why this diagnosis begins after the redirect map has been validated rather than instead of validating it.

Why did a page that was indexed for years fall into this state after the move? Usually because its new URL is competing with a near-duplicate the migration created, or because the new template renders less content than the old one. Both are visible in the two measurements above, and both are fixed on your side rather than by waiting.

Should pages in this state be removed from the sitemap? Only the ones you have decided should not be indexed. Removing a URL from the sitemap does not make the crawler forget it, but it does stop the submitted count overstating what you are asking for — which matters, because a permanently unreachable target trains everybody to ignore the report.

How does this state differ from “Discovered — currently not indexed”? Discovered means the URL is known but has not been fetched, which is a crawl-budget or discovery issue. Crawled means it was fetched and rejected, which is a value judgement about the page. The two have almost nothing in common as problems, and conflating them sends teams to the wrong fix.

Does a client-side-rendered page get judged on the rendered output? It is rendered before the decision is made, but rendering is queued separately and can lag. During that lag the page may be judged on markup that carries almost nothing, which is one reason a client-rendered template shows a larger post-migration spike than a server-rendered one carrying the same content.

Can internal links pull a page out of this state? Sometimes, when the page is genuinely valuable and simply unlinked — an orphaned template is a real and common migration defect. It will not help a page that is thin or duplicated, because those lose on merit rather than on prominence.

How long is too long to wait? Two consecutive weeks with no decline, on a template whose rendered content is intact, whose canonical points at itself, and which is not duplicated elsewhere. At that point the queue explanation is exhausted and something on the page is the reason.

Related

← Back to Index Coverage Handoff