Meta Robots and X-Robots-Tag: noindex, nofollow, and Other Page-Level Directives

Page-level robots directives tell search engines what they may do with a URL after they have fetched it: whether to index it, whether to use its links for discovery, and how much of it to show as a snippet. They work only when the crawler can actually fetch the page. Google’s own specification says these settings “can be read and followed only if crawlers are allowed to access the pages that include these settings” (Google’s robots meta tag specification). That condition sits behind several confusing outcomes in Search Console, and it is the reason crawl control, index control, snippet control and AI control are four separate decisions.

Two delivery methods, one rule set

The same rules can be delivered in two places.

The robots meta tag sits in the <head> of an HTML page:

<meta name="robots" content="noindex, nofollow">

The X-Robots-Tag HTTP header travels with the server response, which makes it the only option for non-HTML files such as PDFs, images or plain-text exports:

HTTP/1.1 200 OK
Content-Type: application/pdf
X-Robots-Tag: noindex

Google’s documentation shows both forms and states that the header name, the user agent name and the values are not case sensitive. Rules written without a user agent apply to all crawlers; rules can also be addressed to one crawler, for example X-Robots-Tag: googlebot: nofollow in the header or <meta name="googlebot" content="nosnippet"> in HTML.

Choose the delivery method by file type and by who controls the deployment. A CMS editor can set a meta tag per page; a server or CDN rule can apply a header to a whole directory of PDFs. Both are read only after a successful fetch.

The rules you will actually use

Google lists the supported values in its specification. The table summarizes what each one does and the limitation to keep in mind.

Two details deserve emphasis. First, nofollow on a page is a request about that page’s outgoing links; it is not a lock on the destination URLs. Second, the all value is the default and, in Google’s words, “has no effect if explicitly listed”, so adding it to templates changes nothing.

Text-level control with data-nosnippet

When only part of a page should stay out of snippets, Google supports the data-nosnippet attribute on span, div and section elements:

<p>Our refund policy is summarized below.
  <span data-nosnippet>Internal approval codes: ...</span>
</p>

The attribute is boolean, so any value you give it is ignored. Google’s featured snippet documentation adds that text marked this way will not appear in featured snippets or regular snippets (featured snippets documentation). If the element is created with JavaScript, add the attribute when the element is first inserted into the DOM, as the specification instructs.

Conflicts: the more restrictive rule wins

Templates, plugins and server rules can stack directives without anyone noticing. Google resolves conflicts with one principle: “the more restrictive rule applies”. Its own example is a page carrying both max-snippet:50 and nosnippet; the result is no snippet at all.

This matters during audits. If a CMS outputs index, follow while an SEO plugin adds noindex, the page is treated as noindexed. If the HTML says index but the HTTP header says noindex, the header still counts. Always inspect the rendered HTML and the response headers together.

Why robots.txt can hide your noindex

One misunderstanding runs like this: a page is disallowed in robots.txt and also carries noindex, and the owner expects it to disappear from Google. It may not.

Google’s block-indexing guide states that for noindex to be effective, the page “must not be blocked by a robots.txt file”, and that if it is blocked, “the crawler will never see the noindex rule, and the page can still appear in search results, for example if other pages link to it” (Google’s block-indexing guide). The robots.txt introduction says the same thing from the other side: robots.txt “is not a mechanism for keeping a web page out of Google” (robots.txt introduction).

The correct sequence for removing an indexed URL from Google is therefore:

1. Leave the URL crawlable.
2. Serve noindex in the meta tag or the header (or return 404/410 if the content is gone).
3. Let Google recrawl the URL and process the rule.
4. Only after the URL has dropped out, decide whether a robots.txt disallow is still useful for crawl management.

Google also states that putting noindex inside robots.txt is not supported.

The decision matrix: name the goal, then the tool

A useful habit is to name which question you are answering before choosing a directive.

Microsoft documents the last row in its October 2026 announcement, alongside a page-level ai-training value. The last four rows are where page-level directives and AI controls get mixed up, so they need precise wording.

Snippet rules and AI features in Google Search

Google’s AI features documentation says that robots.txt rules for Googlebot are the control for access to how sites are crawled for Search, and that to limit the information shown from your pages in Search, site owners can use nosnippet, data-nosnippet, max-snippet or noindex (AI features and your website). In other words, snippet controls shape what Google can show from a page across Search, including AI features, and noindex removes the page from Search entirely.

The Search generative AI control

Since August 31, 2026, Search Console has offered a separate setting under Settings > Search generative AI that lets a property include or exclude its links and content from Search generative AI features. Google’s help page lists AI Overviews, AI Mode and generative AI features in Google Discover as the covered features, and it says the control “isn’t used as a ranking or inclusion signal affecting other parts of Search” (Search generative AI control help).

Three properties of this control are easy to miss:

• It is not Google-Extended. The same help page says the control “doesn’t affect AI training” and points to Google-Extended for that purpose.
• It can be inherited. A child property follows its closest parent property that has stopped inheriting, and by default the top-level domain property. A subfolder property can therefore be excluded or included because of a setting someone else made on the domain property.
• It is not a removal tool. Google states that to block content from appearing in Google Search completely, you use noindex.

Google-Extended

Google-Extended is a robots.txt product token, not a separate crawler. Google’s crawler documentation says it “doesn’t have a separate HTTP request user agent string” and that it “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search” (Google’s common crawlers list). Disallowing it in robots.txt therefore says nothing about AI Overviews in Search; that participation question belongs to the Search generative AI control and to the snippet rules above.

PDFs, legal text and a leaked staging site

The examples below are illustrative configurations, not test results from a specific site.

Example 1: a PDF library that should stay out of results. A site hosts thousands of downloadable PDFs under /files/. Because PDFs have no <head>, the rule is set at the server:

• Preconditions: /files/ is not disallowed in robots.txt.
• Configuration: the server adds X-Robots-Tag: noindex to every response under /files/.
• Expected result: indexed PDFs drop out after they are recrawled.
• Verification: fetch a file with a header inspector and confirm the header is present; check the URL in Search Console’s URL Inspection tool after recrawl.
• Failure condition: if someone later adds Disallow: /files/ before the PDFs are processed, Google stops fetching them and may never see the header.

Example 2: a product page with legal text that should not become a snippet. The page should rank, but the warranty clause should not be quoted in results. Wrap the clause in a div with data-nosnippet. The page stays indexable; only that text is excluded from snippets.

Example 3: a staging site that leaked into the index. The staging host is publicly reachable and was linked from a press preview. Google’s block-indexing guide names password protection as one way to keep a page out of Google, and it suits unreleased content. If password protection is not possible, serve noindex on every staging URL and keep the host crawlable until the URLs drop out. A robots.txt disallow alone would leave already-known URLs eligible to appear without content.

How to test what Google sees

1. Check the response headers. Use curl -I https://example.com/page or your browser’s network panel and look for X-Robots-Tag.
2. Check the rendered HTML. JavaScript can add or change a robots meta tag after load. Google’s specification tells site owners not to add or remove data-nosnippet on existing nodes through JavaScript, to avoid uncertainty from rendering. Keeping robots values stable between the raw and rendered HTML removes the same kind of uncertainty for the meta tag.
3. Use URL Inspection. The indexed result shows whether Google detected noindex; the live test shows whether the current version is indexable, though it cannot confirm how Google’s systems will treat the URL in the future.
4. Read the Page indexing report. URLs that Google could fetch and found noindex on appear under “URL marked ‘noindex'”. URLs blocked by robots.txt appear separately; Google’s help page notes that such blocks are probably intentional and may not need fixing (Page indexing report help).

Mistakes to rule out first

• Using nofollow to hide a page. It controls link following from the page that carries it, not whether the destination can be indexed.
• Combining disallow and noindex during a cleanup. The disallow prevents the noindex from being read.
• Leaving noindex from staging after launch. Google’s site move documentation lists leftover noindex and robots.txt blocks under its “Common mistakes” heading (site moves with URL changes).
• Treating Google-Extended as an AI Overviews switch. It is a training and grounding control for Gemini products, not a Search participation control.
• Assuming one AI setting covers every provider. OpenAI, Anthropic and other providers document their own crawler tokens and purposes; each one has to be read separately.

Choosing the next step

If the URL should not be crawled at all, the question belongs to robots.txt and crawl management. If it should not be shown, use noindex and keep it crawlable. If only part of the page should stay out of results, use the snippet rules. If the concern is Google’s generative AI features, use the Search generative AI control and check whether the property inherits its value. If the concern is training by AI developers, use each provider’s training token, and add the shared training token only as a supplement, since providers honor it from different dates. Naming the goal first is what keeps these controls from cancelling each other out.