Splitting Large Sitemaps for Faster Recrawl

Problem Statement

One sitemap holding 180,000 URLs is valid, submits cleanly, and tells you almost nothing. When the coverage report says 164,000 indexed you cannot tell whether the missing 16,000 are retired archive pages nobody cares about or the entire product catalogue. After a migration that ambiguity is expensive, because the first question anyone asks is which section has not come across. Segmenting the file turns one unreadable number into a per-segment signal. This page sits under Sitemap Re-submission Strategies.

A sitemap index segmented by template One sitemap index file references four child sitemaps split by template — products, categories, articles and archive — each reporting its own submitted and indexed counts. One Index, Four Readable Signals sitemap.xml index of four files products 42k / 41.6k indexed categories 1.2k / 1.2k indexed articles 9k / 8.9k indexed archive 128k / 112k indexed The gap is now attributable: it is the archive, and the catalogue is fine.
The same 16,000-URL shortfall becomes actionable the moment it can be attributed to one segment.

When to Use This Approach

  • The site has more than about 20,000 URLs, or more than one distinct page template.
  • A migration has just completed and you need to know which section is lagging, not just that something is.
  • Different sections have genuinely different priorities — a product catalogue matters more per URL than a 2014 blog archive.
  • You intend to retire part of the estate and want its decline to be visible separately from healthy sections.

Aggregate Numbers Hide the Only Thing You Need

A coverage report on one large sitemap gives you a submitted count and an indexed count. The difference between them is a single number, and a single number cannot be investigated. It could be one broken template, a deliberate noindex on a section you forgot about, an archive that was never worth indexing, or a genuine failure of the redirect map — and every one of those calls for a different response, including “do nothing”.

Teams respond to that ambiguity predictably. They re-submit the sitemap, which changes nothing because the crawler already has it. They request indexing for a handful of URLs, which addresses discovery when the constraint is selection. They open an investigation that spans the whole estate, because there is no way to narrow it. Weeks pass, and the number does not move, because nobody has yet identified which part of the site the number is about.

Segmentation converts that single ambiguous number into several unambiguous ones. It costs one build-time change and it is the difference between “we have lost 9% of our index” and “the 2011–2016 event archive is not being indexed, which is correct, and everything commercial is healthy”. The second statement can be acted on — or deliberately not acted on — the same day.

# The report you want: one line per template, not one number for the site
curl -s https://example.com/sitemap.xml | grep -o '<loc>[^<]*' | sed 's/<loc>//' \
  | while read -r c; do printf '%6s  %s\n' "$(curl -s "$c" | grep -c '<loc>')" "$c"; done

Step-by-Step Instructions

1. Choose the Segmentation Axis

Segment by template, not alphabetically or by arbitrary chunks of 50,000. Templates are what break during a migration — a routing change affects every product URL and no article URLs — so a per-template split maps directly to the thing you will need to fix.

Where revenue concentration is uneven, add a second axis and split high-value URLs into their own file, using the banding from Segmenting High-Value URLs by Revenue Before Migration.

2. Respect the Hard Limits

Each sitemap file holds at most 50,000 URLs and 50 MB uncompressed; an index file references at most 50,000 sitemaps. Aim well below the URL ceiling — 10,000 to 20,000 per file keeps each segment readable and lets you regenerate one segment without touching the others.

# Split a URL list into per-template sitemap files under a shared index
import math, pathlib, xml.sax.saxutils as esc

CHUNK = 15000  # well under the 50k ceiling, keeps each report readable

def write_sitemap(urls, path):
    body = "\n".join(f"  <url><loc>{esc.escape(u)}</loc></url>" for u in urls)
    pathlib.Path(path).write_text(
        '<?xml version="1.0" encoding="UTF-8"?>\n'
        '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">\n'
        f"{body}\n</urlset>\n")

def split(template, urls, out="_site"):
    parts = math.ceil(len(urls) / CHUNK) or 1
    names = []
    for i in range(parts):
        name = f"sitemap-{template}-{i+1}.xml"
        write_sitemap(urls[i * CHUNK:(i + 1) * CHUNK], f"{out}/{name}")
        names.append(name)
    return names

3. Emit the Index File

<?xml version="1.0" encoding="UTF-8"?>
<!-- Absolute URLs only; a relative loc in an index is silently ignored -->
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap><loc>https://example.com/sitemap-products-1.xml</loc></sitemap>
  <sitemap><loc>https://example.com/sitemap-categories-1.xml</loc></sitemap>
  <sitemap><loc>https://example.com/sitemap-articles-1.xml</loc></sitemap>
  <sitemap><loc>https://example.com/sitemap-archive-1.xml</loc></sitemap>
</sitemapindex>

4. Submit the Index, Read the Children

Submitting the index registers every child, and Search Console reports each child separately. That per-child breakdown is the entire point — one row per template, each with its own submitted and indexed count.

Segmentation axis compared Three candidate segmentation axes — alphabetical, fixed chunks and by template — compared on whether a shortfall can be attributed and whether regeneration is cheap. Why Template Beats Every Other Split SPLIT BY ATTRIBUTES A SHORTFALL CHEAP TO REGENERATE Alphabetical no no — reshuffles Fixed 50k chunks no no — reshuffles By template yes yes — one file Any split that reshuffles URLs between files on every rebuild destroys the trend you were watching.
Stability across rebuilds matters as much as attribution — a URL should stay in the same segment for the life of the site.

Worked Example

A marketplace with 180,000 URLs migrated to a new domain and saw its indexed count settle at 164,000 after eight weeks. The single-file sitemap made this look like a broad 9% loss, and three weeks were spent investigating the redirect map for a fault that was not there.

Re-segmenting into four files told a different story within a day of the first crawl:

Segment Submitted Indexed Reading
products 42,000 41,600 healthy
categories 1,200 1,200 healthy
articles 9,000 8,900 healthy
archive 128,000 112,000 the entire shortfall

The archive segment was 2011–2016 event listings for events that had happened. They were correctly redirected, correctly canonical, and simply not worth indexing — a fact nobody could see while they were averaged in with the catalogue. The correct action was to stop treating the number as a defect, mark the archive noindex, and remove it from the sitemap entirely.

Verification

# 1. Index file parses and every child is an absolute URL on the right host
curl -s https://example.com/sitemap.xml \
  | grep -o '<loc>[^<]*' | sed 's/<loc>//' | tee /tmp/children.txt

# 2. Every child sitemap returns 200 and XML
while read -r u; do
  printf '%s ' "$(curl -s -o /dev/null -w '%{http_code} %{content_type}' "$u")"; echo "$u"
done < /tmp/children.txt

# 3. No child exceeds the 50,000-URL ceiling
while read -r u; do
  printf '%6d  %s\n' "$(curl -s "$u" | grep -c '<loc>')" "$u"
done < /tmp/children.txt

If a child sitemap returns HTML, the usual cause is a rewrite rule catching .xml paths — the same class of over-broad rule discussed in Writing Nginx Regex Redirects for Query Strings.

Sitemap size and index limits Three stacked bars showing the 50,000 URL ceiling per file, a recommended 15,000 working size, and the 50 megabyte uncompressed size limit. Limits, and the Size You Should Actually Use Hard ceiling 50,000 URLs per file Recommended 15,000 URLs readable, cheap to regenerate Size ceiling 50 MB uncompressed per file Sitting at the ceiling means any growth forces a reshuffle across files.
Leaving headroom below the ceiling is what keeps segment membership stable as the site grows.

FAQ

Does splitting a sitemap make crawling faster? Not directly — the crawler’s budget is unchanged. What it makes faster is diagnosis, and after a migration that is usually the bottleneck. Knowing within a day that the shortfall is confined to the archive saves weeks of investigating the catalogue.

Should every template have its own file even if it holds ten URLs? Yes, if it is a template you would investigate separately. A ten-URL file costs nothing and gives you an unambiguous row in the report. The exception is a template you never intend to index, which should be left out of the sitemap altogether.

Can I nest a sitemap index inside another index? No. An index may reference sitemaps but not other indexes. If you need more than 50,000 child files, you have a different problem than sitemap structure.

What should I do about URL counts that never reach the submitted total? Decide whether the gap is a defect or a preference. Pages that are deliberately thin, expired, or duplicated will not be indexed no matter how they are advertised, and the right fix is to remove them from the sitemap rather than to keep re-submitting.

Do compressed sitemaps still count against the size limit? The 50 MB limit applies to the uncompressed file, so gzipping saves bandwidth but not headroom. Compression is worth doing for a large archive segment, provided the response carries Content-Encoding: gzip — without it the file is served as unreadable bytes and reports zero discovered URLs.

Should the legacy sitemap be segmented the same way? Yes, and by the same templates. Matching segmentation lets you compare the legacy segment’s falling indexed count against the new segment’s rising one, which is the clearest possible signal that a particular template has finished migrating. Segmenting the two files differently throws that comparison away.

How stable does segment membership need to be? Stable enough that a URL stays in the same file across rebuilds. Membership derived from the template is naturally stable; membership derived from position in a sorted list is not, and a file whose contents reshuffle on every build makes its own trend line meaningless.

Does the index file need to be at the site root? It needs to be reachable and on the same host as the URLs it references, but the path is your choice. Root placement is conventional and makes the Sitemap: line in robots.txt obvious, which is reason enough absent a constraint pushing the other way.

What if one template legitimately has more than 50,000 URLs? Split it into numbered parts within the same template — sitemap-archive-1.xml, sitemap-archive-2.xml — and keep the naming stable. The report then shows several rows for that template, which still aggregates cleanly and still tells you which template the shortfall belongs to.

Is there a downside to many small sitemap files? Only administrative. The index handles up to 50,000 children, and the crawler treats them independently. The real limit is how many rows a person will read in the report, which in practice is a few dozen — beyond that, group by template family rather than by individual template.

Related

← Back to Sitemap Re-submission Strategies