Crawling and indexing: search engine discovery guide

Core definitions: crawling, indexing, ranking

Treat crawling and indexing as two distinct stages. Crawling is discovery and fetching: a crawler visits URLs, fetches the server response, and records resources that affect rendering. Indexing is the evaluation and storage stage: after a page is fetched and (usually) rendered, the search engine decides whether to store the page in its index and make it eligible to appear in search results. Ranking is a separate stage that orders indexed content for queries. Problems at the crawl or index stage prevent a page from ever entering the ranking competition.

How crawling works in practice

When a crawler requests a URL it observes HTTP status codes, redirects, response headers, and the HTML/CSS/JS the server returns. The crawler discovers links (HTML anchors, sitemap entries, hreflang links), embedded resources, and scripts that may produce additional URLs to fetch. Important operational points:

• Crawlers follow internal links and external backlinks as discovery signals.

• Robots.txt is checked before fetching pages; it can prevent crawling but not necessarily indexing if other signals exist.

• JavaScript rendering: many modern crawlers render pages, but blocked or slow resources can stop rendering and hide content. Use server-side rendering or ensure critical content is available in the initial HTML when possible.

• Crawl budget and frequency: search engines prioritise which URLs to revisit. Very large sites should manage indexable URL surfaces to avoid unnecessary crawling of low-value pages.

Indexing: what determines whether a page is stored

Indexing decisions use many signals: robots meta tags, X-Robots-Tag headers, canonical tags, content quality, duplication, and structured data. Being crawled is necessary but not sufficient for indexing — a page can be crawled and excluded from the index if the engine deems it low-value or explicitly instructed to exclude it.

Practical examples of directives that affect indexing:

• HTML meta robots example: <meta name="robots" content="noindex, nofollow">

• Canonical example: <link rel="canonical" href="https://example.com/preferred-page">

• HTTP header example (X-Robots-Tag): X-Robots-Tag: noindex

Verification: how to check crawling and indexing (own site vs third-party)

For pages you own (authoritative checks)

Use Google Search Console's URL Inspection to confirm crawl and index status, view the live render, and see coverage issues for that URL. The Coverage and Pages reports help locate groups of indexation problems. Use the Rich Results Test to validate structured data and see if a result is eligible for enhanced features.

For third-party/publisher pages (external verification)

When you do not control the domain, rely on public signals and live fetches. Use curl and a browser to inspect the HTML the site serves, check HTTP headers, and confirm that the link exists in the page source and rendered DOM. Example commands:

• Get headers only: curl -I https://example.com/page (returns response headers only).

• Fetch HTML as a mobile user-agent: curl -A "Mozilla/5.0 (Linux; Android 12)" https://example.com/page (fetches the full HTML response for that UA).

• Check the rendered DOM with Chrome DevTools Elements panel or a headless renderer to ensure the link is not injected later in a way that search engines might not execute.

Use the site: operator (e.g. site:publisher.com "unique phrase") as a public indexation indicator — it can be useful but is not definitive. For Bing, use Bing Webmaster Tools' Site Explorer to inspect index signals.

Common mistakes and how to fix them

• Robots.txt blocking important resources: robots.txt can prevent crawlers from fetching CSS/JS needed to render content. Solution: allow critical assets or use server-side rendering for core content.

• Accidental noindex or X-Robots-Tag headers: check templates, staging rules, and CDN edge configuration for headers that add noindex. Use curl -I to inspect headers.

• Incorrect canonical usage: pointing many pages to the wrong canonical can remove the preferred pages from the index. Ensure canonical links point to the best single version of content.

• Over-indexed low-value pages (faceted navigation, thin archives): reduce indexable permutations using canonicalization, parameter handling, or purposeful noindex for low-value variants.

• Heavy reliance on client-side rendering without server fallback: ensure critical content appears in the initial HTML or is rendered quickly; otherwise crawlers may not execute the required scripts or may take longer to index the content.

Practical audit checklist

Quick walkthrough to find and fix discovery/indexation issues. Work top-to-bottom on pages or site sections you care about most.

1. Confirm the intended canonical URL is served in HTML and does not conflict with server redirects.
2. Check robots.txt for disallows that affect the page or its CSS/JS assets: curl https://example.com/robots.txt
3. Inspect response headers: curl -I https://example.com/page and look for X-Robots-Tag, status codes, and cache-control values.
4. Validate meta robots in page HTML: ensure you have not left noindex unintentionally.
5. Use URL Inspection (Search Console) for authoritative crawl and index details for pages you own, and run the Live Test to see the current rendered output.
6. Check internal linking: ensure important pages are linked from the global nav, category pages, or contextual content so crawlers can reach them within reasonable depth.
7. Audit duplicate and near-duplicate content: decide which versions should be indexed and canonicalise or use noindex where appropriate.

Context updates for 2026 you should be aware of

Google uses the mobile version as its primary basis for crawling and indexing. Since July 2024, Google crawls sites for Search with Googlebot Smartphone by default. Google also removed traditional cached pages in early 2024, so the 'Cached' view is no longer a troubleshooting resource. Search Generative Experience / AI Overviews are now mainstream in many SERPs; being indexed does not guarantee a page will be surfaced inside AI summaries, which use additional selection and summarisation logic.

Further troubleshooting tips

If pages are not indexed despite being crawlable, review server logs to confirm crawler fetch attempts and response codes. Server logs reveal whether crawlers received 200, 3xx, 4xx or 5xx responses and which user-agents were used. Slow TTFB, frequent 5xx errors, or aggressive rate limits can reduce revisit frequency.

For JavaScript-heavy sites, compare the raw HTML (curl) with the rendered DOM (DevTools) to find content that never materialises without specific user interactions. If critical content requires interactions, surface a server-rendered or pre-rendered version for crawlers.

Quick reference: useful tools

• Google Search Console (URL Inspection, Coverage report).

• Rich Results Test and Schema Markup Validator for structured data.

• Chrome DevTools (Network, Performance, Elements) and headless renderers.

• curl for header and content inspection; server logs for authoritative crawl records; Bing Webmaster Tools Site Explorer for Bing index checks.

FAQ