AI Crawlers and User-Agent Tokens: GPTBot, ClaudeBot, Google-Extended and Which to Block

There is no single “AI bot” to allow or block. Each AI company documents several robots.txt tokens, and each token controls a different use: collecting training data, building a search index, or fetching a page because a user asked a question. Blocking the wrong one can remove a site from an AI search product while doing nothing about training, or the reverse. The practical answer to “which should I block?” is therefore a matrix: provider, token, purpose, and the effect a block is documented to have.

The tokens below are those documented by OpenAI, Anthropic and Google as of November 2026. Google adds two things that are not ordinary tokens: a Search setting that lives outside robots.txt, and a shared AI training signal, based on an IETF draft, that it plans to support in 2027. Anthropic and Microsoft are adopting the same signal on their own dates, covered below.

The matrix

The table summarizes each provider’s own documentation. The column “Documented effect of disallowing” paraphrases the provider’s wording; the sections below quote it.

Two rows need extra care. ChatGPT-User is not a crawler in the automatic sense, and Google-Extended is not a crawler at all.

OpenAI: three independent settings

OpenAI’s bot documentation states that its robots.txt settings are independent of each other, and gives the canonical example: a site can allow OAI-SearchBot to appear in search results while disallowing GPTBot to keep its content out of training (OpenAI’s crawler documentation).

• GPTBot. “Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models”.
• OAI-SearchBot. “Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links”. OpenAI recommends allowing both the token and its published IP ranges for sites that want to appear.
• ChatGPT-User. OpenAI says it “is not used for crawling the web in an automatic fashion”, that “robots.txt rules may not apply”, and that it “is not used to determine whether content may appear in Search”.

OpenAI also notes that it can take about 24 hours after a robots.txt change for its search systems to adjust, and that when both bots are allowed it may use a single crawl for both purposes. A separate agent, OAI-AdsBot, visits only landing pages submitted as ChatGPT ads and, according to OpenAI, is not used for training.

Anthropic: training, search and user retrieval

Anthropic’s help article lists three robots and states that its bots honor robots.txt directives (Anthropic’s crawler documentation).

• ClaudeBot collects content that could contribute to training. “When a site restricts ClaudeBot access, it signals that the site’s future materials should be excluded from our AI model training datasets”.
• Claude-SearchBot crawls to improve search results; disabling it “may reduce your site’s visibility and accuracy in user search results”.
• Claude-User retrieves pages when a person asks Claude a question; disabling it “may reduce your site’s visibility for user-directed web search”.

Anthropic adds three operational details. It supports the non-standard Crawl-delay extension. Blocking its IP addresses instead of using robots.txt “may not work correctly or persistently guarantee an opt-out”, because the block can stop Anthropic from reading the robots.txt file. And browsers that Claude operates on a user’s behalf are documented on a separate page, so agentic browsing should not be assumed to follow the same token as the three robots above.

Google: one crawler, two different controls

Google’s situation differs because AI features are part of Google Search itself.

Googlebot crawls for Search. Google’s AI features guide says robots.txt directives for Googlebot are the control for how sites are crawled for Search, and that nosnippet, data-nosnippet, max-snippet or noindex (page-level directives) limit what is shown from a page in Search (AI features and your website). Blocking Googlebot to avoid AI Overviews would also remove the site from ordinary search crawling.

Google-Extended is a robots.txt product token. Google’s crawler list says it “doesn’t have a separate HTTP request user agent string”, that crawling is done with existing Google user agents, and that it “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search” (Google’s common crawlers list). It governs whether content may be used for training future Gemini models and for grounding in Gemini Apps and Vertex AI products. You will never see “Google-Extended” in a server log, because no request carries it.

The Search generative AI control is not a robots.txt token. It is a Search Console setting, rolled out worldwide as of August 31, 2026, that includes or excludes a site’s links and content from AI Overviews, AI Mode and generative AI features in Discover. Google’s help page says it “doesn’t affect AI training” and points to Google-Extended for that (Search generative AI control help). The setting can also be inherited from a parent property, so a URL-prefix property may be following a choice made on the domain property.

Read together, the answer to the question “Does disallowing Google-Extended remove a site from AI Overviews?” is no. AI Overviews participation is handled by the Search generative AI control and by Search snippet rules; Google-Extended handles training and grounding in Gemini products outside Search.

Other providers

Every provider defines its own tokens. Perplexity, for example, documents PerplexityBot as a crawler that surfaces and links websites in Perplexity search results and states that it is not used to crawl content for AI foundation models, plus a separate Perplexity-User agent for user-initiated visits (Perplexity’s crawler documentation). Before adding any token to robots.txt, read that provider’s current page; tokens and their scopes change, and a token name copied from an old blog post may no longer exist or may mean something else.

The shared AI training signal: adopted on different dates

On October 30, 2026, Google announced plans to support “a shared AI training control in robots.txt”, based on work from the IETF’s AIPREF working group (Google’s announcement). The idea is a shared policy token that expresses a training preference across participating AI providers in one line. Google said that when a site uses the token to opt out of training and does not explicitly name Google-Extended, it plans to treat that as if Google-Extended were disallowed, and that existing company-specific controls will keep working alongside it.

As of November 2026, the shared preference is not enforced the same way by every provider:

• Google describes its announcement as a “draft proposal”, says it aims to support the control “in 2027”, and does not show the token syntax.
• Anthropic says: “As of October 30, 2026, Anthropic has begun recognizing a machine-readable signal that lets website owners express a preference about whether their content may be used to train AI models.” It is “rolling this out across all of our services over time”, and for a given path a robots.txt rule that names ClaudeBot takes priority over the general setting.
• Microsoft published draft robots.txt syntax for the shared preference and says it will honor its new preferences as it crawls, starting February 1, 2027 (Microsoft’s October 2026 announcement).

Microsoft’s example for opting a whole site out of training:

User-Agent: __usage_ai-training
Disallow: /

The token follows an IETF Internet-Draft, not a finished standard, and Anthropic notes that “the way we read and apply these settings may change”. A site that adds it today should keep its provider-specific rules (GPTBot, ClaudeBot, Google-Extended) in place, since those are documented as working now, and recheck each provider’s page for when it starts honoring the shared token.

Microsoft’s separate grounding preference. Microsoft’s announcement also adds a page-level AI Grounding preference, independent of training, set with <meta name="Bingbot" content="ai-grounding:disallow"> or the header X-Robots-Tag: Bingbot: ai-grounding:disallow; the value disallowextended limits grounding to Microsoft-operated experiences. Microsoft says these preferences do not change eligibility for traditional Bing search results, will replace its older NoArchive and NoCache preferences for training and grounding, and that sites currently using NoArchive or NoCache should publish their new preferences before March 1, 2027. It also warns that disallowing grounding “will significantly reduce your content’s visibility in AI-generated experiences”.

Choosing what to block

The decision follows from what the site wants from AI products, not from a list of bot names. Three configurations, shown as illustrative robots.txt fragments:

Allow AI search visibility, decline training.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

Expected effect according to each provider’s documentation: content is withheld from OpenAI and Anthropic training and from Gemini training and grounding, while the site remains eligible for ChatGPT search, Claude search and Google Search including its AI features.

Decline both AI search and training for one provider. Disallow that provider’s search and training tokens. For OpenAI this means GPTBot and OAI-SearchBot; the site can still appear as a navigational link in ChatGPT according to OpenAI.

Stay out of Google’s AI features but stay in Google Search. This is not a robots.txt decision. Use the Search generative AI control in Search Console, check the inheritance setting on every property, and use snippet rules for page-level limits.

Before deploying, check whether a token is user-triggered. Blocking ChatGPT-User or Claude-User affects what happens when a person asks about your page, and OpenAI states its robots.txt rules may not apply to ChatGPT-User at all.

Verifying who is actually requesting

A user-agent string is a claim, not an identity. Anyone can send GPTBot in a header. Verification uses the methods each provider publishes:

• IP ranges. OpenAI publishes JSON files of IP ranges for its agents, linked from its documentation. Anthropic says a source IP on its published list indicates the request comes from Anthropic.
• Reverse DNS for Google. Google documents a reverse and forward DNS check, plus published IP ranges, for verifying Googlebot and other Google crawlers (verifying Google’s crawlers).

A workable check sequence for a suspicious log line:

1. Note the user agent, IP address, timestamp and path.
2. Compare the IP with the provider’s published range or run the provider’s DNS check.
3. If the IP does not match, treat the request as unverified traffic using a borrowed name; robots.txt will not stop it, and firewall or rate-limit rules are the relevant tool.
4. If it matches and the request ignores your robots.txt, collect the evidence and contact the provider through the address on its documentation page.

What robots.txt cannot do here

robots.txt expresses a preference to compliant crawlers. It does not authenticate requesters, remove content already collected, or bind operators who do not publish a policy. Changes also take time to propagate: OpenAI cites roughly a day for search; Google’s robots.txt specification says it generally caches the file for up to 24 hours. For content that must not be reachable, access control (login, paywall, IP restriction) is the mechanism, and each of those choices also affects search crawlers that you may want to keep.

Summary of the decision

Name the use you want to allow or refuse, find the token the provider documents for that use, and verify the effect against the provider’s current page. Training, AI search and user-triggered retrieval are three different permissions at OpenAI and Anthropic. At Google, Search crawling, Search AI participation and Gemini training are three different controls, and only two of them live in robots.txt.