XML sitemap SEO: Crawling & indexation guide

What XML sitemaps do (and don't)

An XML sitemap is a machine-readable file that lists URLs you want search engines to know about. It signals which pages you consider important, provides hints about last modification dates, and groups multiple sitemaps via a sitemap index. Sitemaps speed up discovery and help search engines revisit updated pages more efficiently, but they are guidance rather than commands: search engines still decide which pages to crawl, index, and rank based on quality, relevance, duplication, and site trust.

Put another way: a sitemap helps discovery and indexation hygiene. It does not make thin or duplicate content rank, and it cannot substitute for good site architecture, correct canonicalization, or indexation control (eg. using noindex where appropriate).

How sitemaps interact with crawling, indexing, and ranking

Separate the three processes in your mind: crawling (fetching URLs), indexing (deciding which content to store), and ranking (ordering results). Sitemaps primarily affect crawling and indexation signals: they tell search engines where to start and which pages have changed. They are not a direct ranking factor, but better discovery and fresher indexing can indirectly help pages appear where they otherwise might be missed.

Technical points to note: Google uses the mobile version as its primary basis for crawling and indexing; since July 2024, Google crawls sites for Search with Googlebot Smartphone by default. Also, Google removed the traditional "Cached" page view in early 2024, so the visible cached snapshot is no longer a public troubleshooting surface.

Sitemap mechanics and formats

XML structure and common elements

A standard sitemap uses <urlset> with child <url> entries. Typical child elements are <loc> (the URL), <lastmod> (optional hint of last modification), <changefreq> and <priority> (both optional hints). Search engines may treat these as hints rather than authoritative instructions.

Example entry: <url><loc>https://example.com/page</loc><lastmod>2026-08-01</lastmod></url>

Sitemap indexes, compression and limits

You can group multiple sitemaps using a sitemap index. Per the sitemap protocol specification, a single sitemap file can contain up to 50,000 URLs and should not exceed the uncompressed size limit defined by the spec; when a site exceeds those limits, use a sitemap index to split the set across multiple files. Compressed files (.xml.gz) are supported and commonly used for large sites.

What to include — practical checklist

Include the URLs you want search engines to consider for crawling and indexing. Keep the list focused; avoid adding pages that should not be indexed.

Checklist:

• Canonical, indexable pages with meaningful content (your preferred URLs)
• Pages reachable by internal navigation or important API-driven content you want crawled
• Representative landing pages for pagination, faceted navigation, or category trees when those pages are indexable
• XML sitemap entries that include sensible lastmod values when you have reliable update timestamps

What to exclude

Do not include URLs that should remain out of the index: noindex pages, staging/test environments, internal search result pages, duplicate URLs that point to canonicalized versions, and session or tracking-parameter variants. Also avoid listing non-200 responses, redirects, and soft-404s as primary sitemap entries.

Verification and troubleshooting

Server and file-level checks

From the outside, verify the sitemap file is reachable and served with the correct content-type. Use curl to inspect headers and fetch the file body. For headers only: curl -I https://example.com/sitemap.xml. If you need the file contents, fetch the body with a follow-redirects flag: curl -sL https://example.com/sitemap.xml. Check that the response returns a 200-series status and the Content-Type is XML or application/xml.

Parsing and validation

Validate the XML syntax and confirm URLs are valid. You can pipe the sitemap into an XML parser (for example xmllint) or use online validators. Also use the Google Search Console Sitemaps report to submit and test sitemaps for the properties you control; for each submitted sitemap GSC shows parsing errors, the number of discovered URLs and other diagnostic details.

Indexation checks

For pages you own, use URL Inspection in Google Search Console to see whether Google has crawled and indexed a specific URL and which version it used for indexing. For third-party publisher pages you don’t control, the public site: operator can provide an indication that Google knows a page, but it is not definitive. Combine site: queries with a unique snippet or title fragment to increase confidence, and remember that site: is a public-facing signal, not an authoritative index state.

Log files and crawl budget

If you have access to server logs, inspect crawl behavior after submitting or updating sitemaps. Check whether search engine crawlers request the sitemap and whether they follow links from listed pages. For large sites, a well-maintained sitemap helps ensure crawlers find deep content without wasting budget on duplicate and low-value pages.

Common mistakes and how to avoid them

Mistake: listing non-canonical URLs or pages that return redirects or non-200 statuses. Fix: ensure sitemap entries point to the canonical, indexable URL and remove redirect targets.

Mistake: crowding the sitemap with parameter variants, duplicate parameterized URLs, or session IDs. Fix: use canonical tags, parameter handling, and exclude or normalize variants before listing them in the sitemap.

Mistake: relying on <changefreq> and <priority> to force re-crawls. Fix: treat those elements as optional hints and use reliable lastmod values or push notifications (eg. pubsubhubbub for supported content) to signal changes.

Sitemaps for specific scenarios

Large sites and distributed architectures

Split content into logical sitemap files (by content type, section, or lastmod window) and use a sitemap index. This makes parsing and troubleshooting easier and lets you update only a subset of sitemaps when a vertical of your site changes frequently.

AJAX / client-rendered content

If parts of your site depend on client-side rendering, include server-rendered or pre-rendered canonical URLs in the sitemap so crawlers find usable HTML. Where you can serve content directly to Googlebot Smartphone (per mobile-first indexing practices), ensure the sitemap entries match the URLs that deliver indexable content to users.

Maintenance checklist

Monthly or quarterly:

• Re-submit sitemaps in Google Search Console after major structural changes
• Scan for parse errors or 4xx/5xx responses listed in the sitemap
• Confirm lastmod values are updated for materially changed content
• Review server logs to confirm sitemap fetches by crawlers

Further reading

For related technical topics, see the broader guide on site readiness:

Read the Technical SEO Guide

FAQ