Robots.txt: The Complete Guide to Directives, Testing, and Common Mistakes

A robots.txt file tells crawlers which URLs on a host they may request. It is a crawl control, not an indexing control and not a security measure. Google’s own introduction states it “is not a mechanism for keeping a web page out of Google”, and that a page disallowed in robots.txt “can still be indexed if linked to from other sites” (Google’s robots.txt introduction). The common mistakes share one cause: asking the file to do something it was never designed to do, such as keeping a page out of results or hiding it from people.

Where the file lives and what it covers

The file must be named robots.txt and sit at the root of the host it governs: https://www.example.com/robots.txt applies to https://www.example.com/ and nothing else. Each subdomain and each protocol-host combination needs its own file. Google’s specification sets a maximum file size of 500 kibibytes and says that if the file is not UTF-8 encoded, Google may ignore characters outside that range, which can invalidate rules (Google’s robots.txt specification).

The fields Google supports

Google supports four fields:

The specification states that other fields “such as crawl-delay aren’t supported” by Google. Some other crawlers do read crawl-delay; Anthropic, for example, documents support for it (Anthropic’s crawler documentation). A line Google does not support is ignored by Google, not treated as an error.

Field names are not case-sensitive; path values are. disallow: /Private/ does not block /private/.

Groups: which rules apply to which crawler

Rules are organized in groups, each starting with one or more user-agent lines. Google’s specification says “Only one group is valid for a particular crawler”, chosen by finding the most specific user agent that matches the crawler. Two consequences follow:

1. If a file has a user-agent: * group and a user-agent: Googlebot group, Googlebot follows only its own group. Rules in the * group do not add to it.
2. A crawler with no matching named group follows the * group.

user-agent: *
disallow: /search/
disallow: /cart/

user-agent: Googlebot
disallow: /search/

In this example Googlebot may crawl /cart/, because its own group does not disallow it. If the intent was to block /cart/ for everyone, the Googlebot group must repeat the rule.

Matching: longest path wins, ties go to allow

Within a group, Google uses “the most specific rule based on the length of the rule path”. When rules conflict, “including those with wildcards, Google uses the least restrictive rule”.

user-agent: *
disallow: /docs/
allow: /docs/public/

/docs/public/guide is allowed because the allow path is longer. /docs/internal is disallowed.

Wildcards

Google and other major search engines support a limited form of wildcards in path values:

• * matches zero or more of any valid character.
• $ marks the end of the URL.

user-agent: *
disallow: /*?sessionid=
disallow: /*.pdf$
allow: /downloads/brochure.pdf$

The first rule blocks any URL containing ?sessionid=. The second blocks URLs ending in .pdf; the allow line re-opens one specific file because its path is longer. Without the $, disallow: /*.pdf would also block /whitepapers.pdf-archive/.

Sitemap lines

The sitemap field takes a fully qualified URL and can appear anywhere in the file; it is not tied to a user-agent group. A sitemap listed here is a hint for discovery. Listing it does not guarantee crawling or indexing.

What happens when robots.txt cannot be fetched

Google’s specification describes behavior by response:

• 4xx (except 429): treated as if no valid robots.txt exists, so crawling is unrestricted.
• 5xx and network errors: for the first 12 hours Google stops crawling the site but keeps trying to fetch the file; then it uses the last good version for up to 30 days; after that, if the site is otherwise available, it behaves as if there is no robots.txt.
• Caching: Google generally caches the file for up to 24 hours, longer if refreshing fails.

A misconfigured server that returns 503 for robots.txt can therefore pause crawling of an entire site.

The crawl-or-index decision tree

Before writing a rule, decide which outcome you need.

1. Should the URL be absent from search results?

• Yes, and it is already indexed: keep it crawlable and serve noindex (a robots meta tag or X-Robots-Tag header), or return 404/410 if it is gone. A disallow would hide the noindex from Google; Google’s block-indexing guide says that if a page is blocked by robots.txt “the crawler will never see the noindex rule”.
• Yes, and it must never be seen by anyone: put it behind authentication.
• No: continue.

2. Is crawling the URL wasteful or harmful to the server? (Infinite filter combinations, internal search results, session parameters.)

• Yes: disallow it in robots.txt.
• No: leave it crawlable.

3. Do you need it removed from results quickly?

• Use Search Console’s Removals tool for a temporary block, which Google says lasts about six months (Removals tool help), alongside a permanent method.

Google also states that a noindex rule placed inside robots.txt is not supported.

AI crawlers and the shared training signal

Robots.txt is also where AI providers document their opt-out tokens: GPTBot and OAI-SearchBot for OpenAI, ClaudeBot and Claude-SearchBot for Anthropic, and Google-Extended for Google’s Gemini training and grounding. Each token has its own scope, and blocking a training crawler is not the same as blocking a search crawler.

On October 30, 2026, Google announced plans to support “a shared AI training control in robots.txt”, a common policy token based on IETF AIPREF work that would express a training preference across participating providers (Google’s announcement). Providers are adopting it on different dates. Google aims to support it in 2027. Anthropic says it began recognizing the signal on October 30, 2026 and is rolling it out over time, while treating rules that name ClaudeBot directly as more specific. Microsoft has published draft robots.txt syntax (User-Agent: __usage_ai-training) and says it will honor its new preferences from February 1, 2027 (Microsoft’s October 2026 announcement). The token comes from an IETF Internet-Draft, not a finished standard, so keep provider-specific rules alongside it and recheck each provider’s documentation.

Testing a robots.txt file

1. Fetch it directly. Request /robots.txt and confirm a 200 status, a text/plain content type and the expected content. Check every host and subdomain.
2. Read the robots.txt report in Search Console. It shows which robots.txt files Google found for the top hosts of a property, when each was last crawled, and any warnings or errors.
3. Test individual URLs. URL Inspection reports whether a URL is blocked by robots.txt. For a rule you have not deployed yet, test it with an open-source robots.txt parser before uploading, and confirm the result in URL Inspection after deployment.
4. Check the Page indexing report. “Blocked by robots.txt” lists URLs Google could not crawl. Google’s help notes these are “probably intentionally blocked” (Page indexing report help), so read the list rather than fixing it by default.
5. Keep the previous version. If a deployed rule blocks the wrong URLs, restore the previous file at once. Because Google generally caches robots.txt for up to 24 hours, the correction can take that long to reach Google’s crawlers.

Common mistakes

• Blocking a page to remove it from Google. The URL can stay indexed without content.
• Blocking CSS and JavaScript. If rendering resources are disallowed, Google may not see the page as users do.
• Forgetting that named groups replace the * group.
• Case mismatches between rule paths and real URLs.
• Leaving a staging rule in production. disallow: / copied from a staging site stops crawling of the whole live site. Google’s site move documentation lists “noindex or robots.txt blocks” left over from a migration among its common mistakes.
• Using robots.txt for security. The file is public; it advertises the paths you list.
• Expecting crawl-delay to slow Googlebot. Google does not support it.

A worked example

As an illustration, take an online store with faceted navigation that generates millions of filter URLs, an internal search page, and a few PDFs that should stay out of results.

user-agent: *
disallow: /search?
disallow: /*?*color=
disallow: /*?*size=

sitemap: https://www.example.com/sitemap-index.xml

The search rule ends in ? on purpose. A rule path matches every URL path that starts with it, so disallow: /search would also block /search-guide or /searchable-products. In Google’s robots.txt syntax ? is an ordinary character (only * and $ are wildcards), so /search? reaches only the search results URLs that carry a query string. If the site’s search results use a path such as /search/boots, add disallow: /search/ as well, after checking the real URL structure.

There is deliberately no allow: /*?category= line. Google applies the most specific matching rule, measured by the length of the rule path, so allow: /*?category= (12 characters) would beat disallow: /*?*color= (10 characters) and reopen every filtered URL that carries a category parameter. Category URLs without filters need no allow rule, because no disallow rule matches them.

Test matrix for the file above:

– Preconditions: category pages without filters are the canonical landing pages; filter combinations add no unique content worth indexing.

• Expected result: crawlers spend requests on category and product URLs instead of filter combinations.
• Verification: check that the robots.txt report shows the new version fetched and that URL Inspection on a filtered URL reports “Blocked by robots.txt”; then watch Crawl Stats for requests to parameter URLs over the following weeks.
• PDFs: not handled here. Serve X-Robots-Tag: noindex on them and keep them crawlable, since a disallow would stop Google from seeing the header.
• Limitation: URLs already indexed under the blocked patterns may stay in the index without content until they are dropped or handled with noindex before blocking.

Robots.txt does one job well: it tells compliant crawlers where not to go. Pair it with noindex, status codes, canonical tags and authentication for the jobs it does not do.