Robots.txt SEO: control crawling efficiently

What robots.txt does — and what it doesn’t

Robots.txt is a plain-text file placed at a site’s root (https://example.com/robots.txt). Its role is narrowly scoped: it gives crawling instructions to well-behaved crawlers. Use it to reduce unnecessary fetches on low-value areas and to improve crawl efficiency; do not rely on it to control indexation or to hide sensitive content.

Controls crawling, not indexation

A Disallow directive prevents compliant crawlers from fetching a URL path. If a page is blocked from crawling, search engines may still index its URL based on external signals (for example, links) but they won’t see the page’s HTML or meta robots tags. To prevent indexing reliably, allow crawling so the crawler can read a meta robots noindex tag or use an X-Robots-Tag response header with noindex on the resource itself.

Public, advisory, and not a security control

Robots.txt is publicly accessible and advisory: any user can read it at /robots.txt, and malicious crawlers may ignore its directives. Do not list secrets or private paths in robots.txt; treat it as a crawl-control document, not an access-control mechanism.

Core syntax and common directives

Most sites will use a small set of directives. The specification and practical extensions used by major search engines support basic patterns, wildcards and sitemap pointers. Keep rules simple and document intent in comments when possible.

Key directives (examples shown as literal lines):

• User-agent: <bot-name> — target one crawler; use User-agent: * for all crawlers.
• Disallow: /path/ — prevent fetching paths under /path/.
• Allow: /path/file.js — explicitly permit a path under a disallowed parent (supported by major engines).
• Sitemap: https://example.com/sitemap.xml — points crawlers to your XML sitemap(s).
• Wildcards: * (match any sequence) and $ (end-of-string) are supported in practice by major engines; use them carefully for query-parameter patterns.
• Non-standard directives: Crawl-delay and Host are recognized by some crawlers but are not part of the original standard — test behavior for the user-agents you care about.

How robots.txt interacts with indexing and ranking

Separate crawling, indexing and ranking: robots.txt affects the crawling stage (whether a crawler fetches a URL). Indexing requires the crawler to read page content or a meta/X-Robots-Tag. Ranking is a downstream process that relies on signals, some of which come from page content; if a crawler cannot fetch a page, those content-derived signals are unavailable.

Practical consequences:

• Blocking a page in robots.txt means the search engine cannot fetch the page to read meta robots noindex tags or structured data.
• If an important resource (CSS/JS) is blocked, mobile-first indexing and rendering can fail to see the page correctly; since Google uses the mobile version as its primary basis for crawling and indexing, and since Googlebot Smartphone is the default crawler as of July 2024, allow resources required for rendering on mobile.

Verification: how to test what crawlers actually see

Verify robots.txt accessibility and content from the outside and confirm how your site responds to real crawler requests. Use server logs, direct fetches and Search Console tools for pages you control.

Quick checks with curl

To fetch HTTP response headers for robots.txt (headers only):

• curl -I https://example.com/robots.txt — returns response headers only; verify status code and Content-Type.

To retrieve the full file as a specific user-agent (HTML and directives):

• curl -A "Googlebot" https://example.com/robots.txt — returns the file body using the specified user-agent string.

Server logs and real-crawl evidence

Inspect server logs for requests to /robots.txt and for crawler user-agents like Googlebot or Bingbot. Logs show fetch frequency, response codes, and whether crawlers attempted disallowed URLs. For pages you own, use Search Console’s URL Inspection to see crawl and index signals; for third-party pages, a site: query gives an indication but is not definitive.

Common mistakes and how to fix them

Avoid these frequent errors when managing robots.txt.

• Blocking CSS/JS needed for rendering: fixes — allow paths for assets required by the mobile rendering pipeline so Googlebot Smartphone can render pages accurately.
• Expecting robots.txt to noindex URLs: fixes — remove the Disallow for the page and use a meta robots noindex or X-Robots-Tag header so search engines can see the instruction.
• Listing sensitive URLs in robots.txt: fixes — do not expose private paths in robots.txt; protect them via authentication and proper server-side access controls.
• Overly complex wildcard rules that unintentionally match valid pages: fixes — test each pattern with sample URLs and document intent in comments, keeping rules as specific as possible.

Recommended robots.txt strategies

Adopt a conservative, testable approach: keep robots.txt minimal, point crawlers to sitemaps, and prefer page-level meta/X-Robots-Tag controls for index management.

Practical patterns:

• Default allow plus sitemap: include User-agent: * and a Sitemap directive so crawlers can discover your structure quickly.
• Disallow low-value query pages or internal search results, but avoid blocking parameter patterns that also match canonical content; prefer parameter handling in your sitemap or via canonical tags.
• Staging or development: block crawlers while staging is public, but remove or change the block before launch; don’t rely on robots.txt as the only safeguard for pre-production assets.

Examples you can adapt

Simple site with sitemap

User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xml

Common CMS (allow admin-ajax)

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap_index.xml

Staging block (public server but not yet live)

User-agent: *
Disallow: /

Warning: staging blocks must be removed prior to launch; also secure staging with authentication so search engines and third parties cannot accidentally index it if the robots file is changed.

Maintaining robots.txt at scale

For large sites, treat robots.txt as a configuration artifact: keep it in source control, review changes in pull requests, and deploy with the site so staging and production have correct, environment-specific files. Automate tests that fetch the deployed robots.txt and validate rule syntax against representative URLs.

If you operate multiple subdomains, remember that robots.txt is host-specific: rules at example.com/robots.txt do not apply to sub.example.com.

FAQ