how to use robots.txt file for SEO

What this guide covers

This post explains what robots.txt does and does not control, shows correct syntax and practical examples, gives command-line checks you can run, and lists troubleshooting steps to recover from common mistakes. You’ll also get verification tips that work in 2026 search environments (mobile-first indexing is Google’s default since July 2024 and Google removed traditional cached pages in early 2024).

What robots.txt is and what it controls

robots.txt is a plain text file at the root of a site that communicates crawling instructions to well-behaved web robots. It is part of the Robots Exclusion Standard. Use the file to tell crawlers which paths they may fetch; it controls crawling, not indexing or ranking.

Key distinctions to keep in mind:

Crawling vs indexing vs ranking — robots.txt affects crawling (what an agent fetches). A URL blocked from crawling can still be indexed if other pages link to it and the search engine chooses to index the URL without fetching its content.

robots.txt is advisory — only compliant crawlers will follow it. Malicious bots, spam harvesters and some scanners may ignore it.

robots.txt does not replace access controls. Use HTTP authentication, password protection, or server-side controls for private content.

Where robots.txt must live and basic rules

robots.txt must be served from the site root on the exact host and protocol for which the rules apply. Example locations that are distinct: https://example.com/robots.txt and https://sub.example.com/robots.txt. You cannot place a robots.txt in a subfolder and expect it to control the whole domain.

For convenience and examples in this guide, the canonical test path is http://www.yoursite.com/robots.txt.

robots.txt syntax and common directives

A minimal robots.txt uses directives grouped by user-agent. Lines are case-insensitive for directive names. Examples below use the standard tokens supported by most major crawlers:

Block all crawlers from a directory:

User-agent: *

Disallow: /private/

Allow a specific file while disallowing its directory (useful for Googlebot-style parsing):

User-agent: *

Disallow: /images/

Allow: /images/logo.png

Sitemaps can be declared in robots.txt to point crawlers at your XML sitemap:

Sitemap: http://www.yoursite.com/sitemap.xml

Pattern matching: major search engines support the asterisk (*) wildcard and the end-of-string dollar ($) in robots.txt patterns. The non-standard Crawl-delay directive is ignored by some engines (Google does not support Crawl-delay).

Practical examples

1) Allow all crawlers to fetch everything (default safe option):

User-agent: *

Disallow:

2) Block a single crawler (e.g., disallow a specific agent from all content):

User-agent: BadBot

Disallow: /

3) Prevent crawlers from fetching resources required for rendering (common mistake — see troubleshooting):

Disallow: /static/js/

Blocking CSS/JS can stop Google from seeing your page as users see it. Since Google uses the mobile version as its primary basis for crawling and indexing and evaluates Core Web Vitals from the rendered page, do not block resources required for rendering.

How to verify and troubleshoot robots.txt

Basic checks you can run from outside the site:

Fetch the file body: curl http://www.yoursite.com/robots.txt — this returns the file content a crawler would see.

Check headers only: curl -I http://www.yoursite.com/robots.txt — verifies HTTP status (200 vs 4xx/5xx) and Content-Type. -I returns response headers only.

View how browsers receive it: open https://www.yoursite.com/robots.txt in a browser and confirm the same content as curl.

Inspect server logs for real crawler requests to /robots.txt and for Googlebot or other agents attempting to fetch blocked pages — logs are the most reliable third-party signal of crawler behavior.

Checks you should run when you own the property:

Use Google Search Console’s URL Inspection for individual pages to see if Google has fetched or indexed a URL. URL Inspection is authoritative for properties you own.

After changing robots.txt, monitor server logs and the Performance report in Search Console to confirm crawl behavior for important pages.

Common signs of problems and what to do

You accidentally blocked CSS/JS — symptom: pages render differently to crawlers; Solution: remove resource paths from Disallow so crawlers can fetch them.

robots.txt returns 404/5xx — symptom: some crawlers may treat the site as fully crawlable or suspend crawling temporarily; Solution: restore a valid robots.txt that returns 200 and correct Content-Type.

You blocked an entire host or protocol by placing robots.txt on the wrong subdomain — symptom: pages vanish from crawl logs; Solution: add robots.txt to the correct host (and check redirects).

If a page is blocked from crawling but still appears in search results without a snippet, that indicates the URL was indexed from external signals even though the content was not fetched. To remove such URLs from results, use proper HTTP authentication, remove the page, or serve a noindex directive on the page itself (noindex requires the crawler to fetch the page, so do not block it in robots.txt if you plan to use noindex).

Checklist: deploy and review robots.txt safely

Place robots.txt at the site root for the exact host and protocol.

Test the file with curl (body and headers) and in a browser.

Avoid blocking CSS, JavaScript, or other assets used for rendering unless you intentionally want to prevent rendering by crawlers.

Announce sitemaps in robots.txt to help crawlers discover your indexable URLs.

Monitor server logs and Search Console for crawl behavior after changes.

If you need a step-by-step reference, read Read the Technical SEO Guide for broader context on crawlability and Core Web Vitals.

Additional considerations in 2026

Because Google uses the mobile version as its primary basis for crawling and indexing and because traditional cached pages were removed in early 2024, content intentionally hidden from crawlers via robots.txt may be less likely to surface in modern AI-driven overviews and Search Generative Experience features. If you want content to be available to AI overviews or to be used for rich SERP features, allow crawlers to fetch and index the page and use structured data where appropriate.

FAQ