Robots.txt: what it is and how it works

What is robots.txt?

robots.txt (the Robots Exclusion Protocol) is a plain-text file located at the root of a host — e.g. https://example.com/robots.txt — that gives simple crawl instructions to user-agents (web crawlers). It is public and machine-readable; it does not authenticate or hide content. Crawlers read robots.txt to learn which paths the site owner prefers they avoid or prioritise.

Why robots.txt matters for SEO

Robots.txt controls crawling. Crawling is the discovery and fetching stage of how search engines interact with your site; if a crawler is blocked from fetching a URL, it may not see on-page directives (like meta robots) or page content needed to be indexed. That means robots.txt can indirectly influence indexation. It does not directly set ranking order — ranking uses many signals after indexing. Use robots.txt to protect server resources, avoid crawling duplicated or internal tooling pages, and make sure crawlers can access assets required for correct rendering.

How robots.txt works

A crawler fetches /robots.txt before it requests other pages on the same host and applies the first matching User-agent stanza and its directives. Common directives supported by major crawlers include User-agent, Disallow, Allow and Sitemap. Support for pattern matching and additional directives varies by crawler—Google recognises patterns such as * and $ and respects Allow/Disallow precedence rules per its documentation.

Typical directives and examples

A minimal robots.txt example:

User-agent: *
Disallow: /private/
Allow: /public/
Sitemap: https://example.com/sitemap.xml

Notes: place robots.txt at the site root. If robots.txt returns an HTTP 404, most crawlers treat that as 'no restrictions'; if it cannot be retrieved or returns server errors, crawler behaviour can vary—test and monitor to avoid accidental broad blocking. For Google-specific behaviour and examples, consult Google's robots.txt documentation.

Types of robots.txt

Different patterns of robots.txt are used depending on the goal. Below are common approaches with concise pros and cons.

- Site-wide block (Disallow: /)
Pros: immediately prevents crawler access to the whole host.
Cons: prevents crawlers from fetching content and assets; can stop indexation of public pages.

- Path-specific rules (Disallow: /private/)
Pros: easy to exclude internal or admin paths.
Cons: robots.txt is public; do not list sensitive paths you expect to keep secret.

- User-agent specific rules (User-agent: Googlebot)
Pros: tailor behaviour for specific crawlers.
Cons: increased maintenance; some crawlers ignore non-standard directives.

How to get started with robots.txt

1) Decide what needs protection from crawling (server-heavy areas, staging paths) and what must remain available to crawlers (public pages, CSS/JS required for rendering).
2) Create a plain-text file named robots.txt at the site root. Test locally and on a staging host before deploying to production.
3) Deploy and verify using the technical checklist below. Keep the file simple and document any User-agent-specific rules so future maintainers understand intent.

Common robots.txt mistakes

• Relying on robots.txt to hide sensitive data — robots.txt is public and should not be used for security. Use authentication or remove the resource.
• Blocking CSS/JS needed for rendering — this can harm how search engines understand a page’s layout and content.
• Blocking pages that contain meta noindex — if Googlebot is blocked from fetching the page, it cannot see the meta noindex directive.
• Placing robots.txt under a subpath (e.g. /blog/robots.txt) — crawlers look only for the root-level file.
• Syntax errors and wrong HTTP status codes — ensure the file is served with an appropriate 200 response.

Robots.txt verification: technical checklist

Use the checks below after deploying or updating robots.txt. Each item lists where to verify and a clear pass condition.

**File reachable** — where to verify: curl -I https://example.com/robots.txt — passes when the request returns HTTP 200 and the file is the expected content.
**Correct directives** — where to verify: curl https://example.com/robots.txt or view-source in the browser — passes when the User-agent/Disallow/Allow lines match your policy.
**Not blocking rendering assets** — where to verify: Chrome DevTools Network and Coverage or fetch via a crawler simulator — passes when CSS/JS required for rendering are not disallowed.
**Google-block check (own site)** — where to verify: Google Search Console URL Inspection — passes when Inspection shows the URL is crawlable and not "Blocked by robots.txt".
**External crawl check** — where to verify: use curl or a third-party crawler to request a blocked URL and confirm it returns 200 for the page but the crawler was denied access — passes when behaviour matches expectations.
**Sitemap presence** — where to verify: robots.txt content and Google Search Console Sitemaps report — passes when the sitemap URL is listed in robots.txt or submitted in Search Console and accepted.

Tools and commands to verify robots.txt

curl (headers only): curl -I https://example.com/robots.txt — returns HTTP status and response headers. curl (full file): curl https://example.com/robots.txt — prints content for inspection. Chrome DevTools: open the file URL or view page network requests to confirm assets are allowed. Google Search Console URL Inspection: for a URL you own, the tool reports if Google sees the page and whether it was blocked by robots.txt. Bing Webmaster Tools Site Explorer can show how Bing handled the file for sites you verify. Use server logs to confirm which user-agents requested robots.txt and how often.

For authoritative details about supported directives and precedence, consult the official documentation at https://developers.google.com/search/docs/advanced/robots/intro

Read the Technical SEO Guide

Frequently asked questions

Build authority with quality backlinks