Roughly half of news publishers now block at least one AI crawler in robots.txt, according to a July 2026 crawl of nearly 11,000 domains (HasData). The same study found that among domains Google's AI Mode actually cited, 52 percent had an AI block in place. Blocking and being cited are not opposites.
That is because there is no such thing as "AI crawlers" as a single category. Every major vendor now runs at least two agents with different jobs, and the one that trains a model is not the one that fetches your page to answer a question. Block the wrong one and you opt out of the traffic while the training carries on.
This is the reference we use with clients. Every user agent below was checked against the vendor's own documentation in September 2026, and we have flagged the two places where the documentation and the observed behaviour disagree.
Three classes of AI bot, and why the distinction matters
The mental model that makes robots.txt decisions easy is to sort every AI agent into one of three buckets.
Training crawlers fetch pages in bulk to build datasets. Blocking one means your future content is left out of the next model. It does not affect what the assistant can find today.
Search-index crawlers build a retrieval index the assistant queries at answer time. Blocking one removes you from the pool of pages the assistant can cite. This is the bucket that decides whether you appear in ChatGPT search, Claude's web results or Perplexity.
User-triggered fetchers retrieve a specific URL because a person asked the assistant to read it, summarise it or check it. Several vendors say these fetchers may ignore robots.txt because a human requested the page, the same way a browser does.
The pattern that has emerged across OpenAI, Anthropic, Google, Apple and Meta is that training is opt-out by robots.txt, search is opt-out by robots.txt, and user fetching is often not controllable there at all. Once you see that, most of the "should I block AI" debate collapses into "do I want to be cited". If the answer is yes, you block training and nothing else.
The verified AI user agent list
Each row links to the page we checked it against. "Honours robots.txt" is the vendor's own claim; see the notes below the table for where that claim is disputed.
| Vendor | User agent token | Class | Used for training | Honours robots.txt | What blocking it does |
|---|---|---|---|---|---|
| OpenAI | GPTBot | Training | Yes | Yes | Excludes future content from OpenAI model training |
| OpenAI | OAI-SearchBot | Search index | No | Yes | Removes you from ChatGPT search results and citations |
| OpenAI | ChatGPT-User | User fetch | No | "Rules may not apply" | Limited; fetches happen when a user asks ChatGPT to open your page |
| Anthropic | ClaudeBot | Training | Yes | Yes | Excludes future content from Anthropic model training |
| Anthropic | Claude-SearchBot | Search index | No | Yes | Reduces visibility and accuracy in Claude search results |
| Anthropic | Claude-User | User fetch | No | Yes | Stops Claude retrieving your page when a user asks |
| Googlebot | Search index | Feeds Search, incl. AI Overviews and AI Mode | Yes | Removes you from Google Search and every AI feature inside it | |
| Google-Extended | Training and grounding control | Yes (Gemini apps and Vertex AI) | Yes | Opts out of Gemini training and grounding; no effect on Search or AI Overviews | |
| Perplexity | PerplexityBot | Search index | No | Yes | Removes you from Perplexity results |
| Perplexity | Perplexity-User | User fetch | No | "Generally ignores robots.txt" | Little in practice; see the Cloudflare dispute below |
| Apple | Applebot | Search index | Yes, unless Applebot-Extended is disallowed | Yes | Removes you from Siri, Spotlight and Safari search |
| Apple | Applebot-Extended | Training control | Controls training use of Applebot data | Yes | Opts out of Apple foundation model training; still eligible for search |
| Meta | meta-externalagent | Training and indexing | Yes | Yes | Excludes content from Meta AI training and direct indexing |
| Meta | meta-externalfetcher | User fetch | No | "May bypass robots.txt" | Little in practice |
| Meta | meta-webindexer | Search index | No | Yes | Removes you from Meta AI search results |
| Amazon | Amazonbot | Mixed: Alexa and Amazon AI training | Yes | Yes (no crawl-delay) | Excludes you from Alexa answers and Amazon model training |
| ByteDance | Bytespider | Training and Toutiao search | Yes | Claimed; widely disputed | Nominal; enforce at the firewall |
| Common Crawl | CCBot | Open archive | Indirectly: the archive is downloadable by anyone | Yes | Keeps you out of future Common Crawl snapshots |
| Microsoft | bingbot | Search index | Feeds Bing, Copilot and ChatGPT search | Yes | Removes you from Bing, Copilot and a large share of ChatGPT citations |
| Mistral | MistralAI-Training | Training | Yes | Yes | Excludes content from Mistral model training |
| Mistral | MistralAI-Index | Search index | No | Yes | Removes you from Mistral's search answers in Vibe |
| Mistral | MistralAI-User | User fetch | No | Yes | Stops Mistral fetching your page on request |
| DuckDuckGo | DuckAssistBot | Real-time answer fetch | No | Yes (72 hour lag) | Opts out of DuckAssist answers; no effect on DuckDuckGo rankings |
| Cohere | None active | n/a | Cohere says it does not currently crawl to train | n/a | Nothing to block yet |
A few notes on the rows that trip people up.
OpenAI's docs state that each agent's setting is independent, so allowing OAI-SearchBot while disallowing GPTBot is the intended configuration for "cite me, don't train on me" (OpenAI). GPTBot and OAI-SearchBot are both on version 1.4 at the time of writing, and OpenAI has added an OAI-AdsBot that validates landing pages submitted as ChatGPT ads. We covered what ChatGPT actually reads once it has fetched a page in ChatGPT's index and cache.
Anthropic publishes verified IP addresses at claude.com/crawling/bots.json and says all three agents respect robots.txt, including Claude-User, which is stricter than the equivalent fetchers from OpenAI and Perplexity (Anthropic).
Apple's setup is unusual. Applebot-Extended never crawls anything. It is a switch that tells Apple whether the pages Applebot already fetched may be used to train its foundation models, and Apple states that pages disallowing it can still appear in search (Apple). We wrote up the wider crawl expansion in Applebot and Siri's AI search.
Amazon is the one vendor that has folded training into its main crawler. Amazonbot's documented purpose includes training Amazon's AI models, and the docs note it ignores crawl-delay (Amazon).
Microsoft's role is bigger than its own products. Bingbot builds the index that powers Copilot, and ChatGPT search launched on it: Seer Interactive found 87 percent of ChatGPT search citations matched Bing's top organic results in early 2025 (Seer Interactive). By August 2025 Semrush had evidence that paid ChatGPT tiers also pull from Google (Semrush), so the honest description is a blended stack. Blocking bingbot to save crawl budget is a mistake we still see on large sites.
Google-Extended is not the AI Overviews switch
This is the single most common misunderstanding we see in client robots.txt files, so it gets its own section.
Google-Extended controls whether your content is used to train future Gemini models and for grounding in Gemini apps and the Vertex AI API. Google's crawler documentation says in plain words that it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal" (Google).
AI Overviews and AI Mode are Search features. They are built from what Googlebot indexes, and Google's AI features documentation says the way to limit what appears in them is the standard set of controls: nosnippet, data-nosnippet, max-snippet and noindex (Google). The same page says there are no extra requirements or files needed to appear in them.
So disallowing Google-Extended is a clean way to say "do not train Gemini on this" with no search cost. It does nothing to your AI Overviews exposure in either direction. If you are weighing the trade-off of trying to limit that exposure, we ran through the real options in should you opt out of AI Overviews and AI Mode.
The Perplexity dispute, and what it tells you about enforcement
In August 2025 Cloudflare published evidence that Perplexity was fetching pages from sites that had blocked PerplexityBot and Perplexity-User, using an undeclared user agent that impersonated Chrome on macOS, rotating through IP addresses outside its published ranges, and switching autonomous system numbers to get around firewall rules. Cloudflare tested it with new, unlinked honeypot domains that only Perplexity could have reached by crawling, delisted Perplexity as a verified bot, and added rules to block the behaviour (Search Engine Land).
Perplexity's response was that its requests are made on behalf of users in real time, are closer to a browser than a crawler, and should not be governed by crawler rules. Its own documentation still says Perplexity-User "generally ignores robots.txt rules" for the same reason (Perplexity).
Bytespider has the same shape without the public row. ByteDance's webmaster documentation, hosted at zhanzhang.toutiao.com and not reliably reachable from outside China, says it honours robots.txt. Fortune reported in 2024 that it ignores the file, and TollBit research in August 2026 found it reaching disallowed pages on close to half the European sites naming it (PPC Land).
Robots.txt still earns its place. The major vendors do follow it, and it remains the only signal that tells a compliant crawler what you want. Treat it as a published preference rather than a lock. If you need a real block, do it at the CDN or firewall using the vendor's published IP lists, and keep robots.txt as the statement of intent that sits on top.
Cloudflare, pay per crawl and the new defaults
Cloudflare, which fronts around a fifth of the web, changed the default in July 2025. New domains block AI crawlers unless the owner turns that off, and a private beta of Pay Per Crawl let sites answer a crawler with HTTP 402 Payment Required and a per-request price, using signed requests so a bot cannot spoof its identity (Cloudflare, Cloudflare).
That was the moment "block AI crawlers" stopped being a robots.txt question for a lot of sites, because the block happened before the file was read. We looked at the visibility consequences at the time in Cloudflare blocking AI crawlers and SEO visibility.
Two things have moved since.
In July 2026 Cloudflare shifted from charging per crawl to models that pay publishers when content is actually cited or served in an answer, working with Ceramic.ai on a pay-per-query arrangement and with You.com on on-demand purchase of specific pages. Cloudflare's reasoning was that a page can be crawled once and cited thousands of times, or crawled constantly and never cited, so crawl counts were a bad proxy for value (PPC Land).
At the same time it launched Bot Preference Sync, which rewrites your robots.txt from three dashboard settings, Search, Agent and Training, and keeps the user agent lists current from its bot directory. Sites that say they monetise pages with ads get Training set to Disallow by default at onboarding; everyone else starts open (Cloudflare). If you are on Cloudflare, check the dashboard before you touch the file, because the file may now be generated for you.
Content Signals, aipref and llms.txt: what is real
There are three "standards" people ask about. They are at very different stages.
Cloudflare's Content Signals Policy, announced in September 2025, adds a Content-Signal: search=yes, ai-input=no, ai-train=no line to robots.txt and was rolled out to more than 3.8 million managed domains with search allowed and training disallowed by default (Cloudflare). Cloudflare frames it as a reservation of rights under Article 4 of the EU copyright directive. Google's John Mueller was blunter in July 2026: it "has no effects whatsoever for any crawler or LLM" and only adds maintenance to the file (Search Engine Roundtable). Treat it as a legal marker and expect no crawler to change its behaviour because of it.
The IETF's AI Preferences working group is building the version that could become a real standard. Its two drafts, draft-ietf-aipref-vocab and draft-ietf-aipref-attach, define a vocabulary (training, AI search, AI input) and a way to attach it to content via robots.txt, well-known URIs and HTTP headers, with both due to the IESG by 31 August 2026 (IETF). Notes from the April 2026 meeting describe the vocabulary as close but the AI input term still being argued over (EDRLab). Check the datatracker for whether the deadline held. Until vendors implement it, it changes nothing in your file.
llms.txt is the one with the most tooling and the least effect. Mueller said in June 2025 that no AI system used it and that server logs showed no bot requesting it (Search Engine Roundtable), Google's AI features page says no extra files are needed, and his July 2026 comment confirmed Google still ignores it. Our position has not changed since llms.txt: useful standard or SEO theatre. It costs nothing, so generate one with the llms.txt generator if a client wants it, but never let it stand in for a working robots.txt.
Three robots.txt policies you can paste in
Pick the one that matches your goal. All three keep Googlebot and bingbot open, because nothing below is worth losing organic search over.
Policy 1: allow everything
This is the right default for most businesses that want to be found. You do not need to list AI agents at all; the wildcard covers them.
User-agent: *
Disallow:
Sitemap: https://www.example.com/sitemap.xml
Policy 2: block training only, stay visible in AI answers
The one we recommend to most content-led clients. Training crawlers are disallowed. Search-index crawlers and user fetchers are explicitly allowed so that a future change to the wildcard cannot silently remove them.
# Training crawlers: opt out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: MistralAI-Training
User-agent: CCBot
User-agent: Bytespider
User-agent: Amazonbot
Disallow: /
# Search and answer agents: keep open
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: meta-webindexer
User-agent: MistralAI-Index
User-agent: MistralAI-User
User-agent: DuckAssistBot
User-agent: Applebot
User-agent: Googlebot
User-agent: bingbot
Allow: /
User-agent: *
Disallow:
Sitemap: https://www.example.com/sitemap.xml
Two judgement calls are baked in. Amazonbot sits in the training group because Amazon does not offer a separate training token, so blocking it also costs you Alexa answers; move it if that matters to you. Applebot-Extended is disallowed while Applebot is allowed, which is exactly the split Apple designed.
Policy 3: block everything AI, keep classic search
For paywalled, proprietary or regulated content where you have decided the citation is not worth the exposure. Understand that this removes you from ChatGPT search, Claude, Perplexity, Meta AI and DuckAssist, but not from Google's AI Overviews or AI Mode, which can only be limited with snippet controls.
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: meta-externalagent
User-agent: meta-externalfetcher
User-agent: meta-webindexer
User-agent: Amazonbot
User-agent: Bytespider
User-agent: CCBot
User-agent: MistralAI-Training
User-agent: MistralAI-Index
User-agent: MistralAI-User
User-agent: DuckAssistBot
Disallow: /
User-agent: *
Disallow:
Sitemap: https://www.example.com/sitemap.xml
Whichever you choose, test it before you deploy. Our robots.txt tester shows which line decides each URL for a given user agent, and the robots.txt generator has a block-AI-crawlers toggle that writes the training group for you without touching your wildcard rules. The mistake that costs the most is a stray Disallow: / under User-agent: *, which takes you out of Google entirely, and it is the first thing we check in any technical SEO audit.
How to decide, and when to revisit
The decision is simpler than the list makes it look. If you sell something and want to be recommended, run policy 2 and accept that training happens on the pages you leave open to search agents anyway. If your content is the product, run policy 3 and enforce it at the edge, because the sites that most need the block are the ones the stealth crawlers target.
Then revisit it quarterly. Vendor agents change version numbers and purposes without fanfare, Cloudflare's defaults have moved twice in a year, and the IETF work may finally give you a single vocabulary instead of twenty user agent tokens. Watch your logs for agents that are not on this list, and check them against the vendor docs linked above rather than a third-party directory.
If you would rather have someone go through your robots.txt, your CDN settings and your server logs together and tell you which AI surfaces you are actually visible in, that is part of our technical SEO work, and a free SEO review is the place to start.
Frequently asked questions
Does blocking Google-Extended remove my site from AI Overviews?
No. Google-Extended only controls whether your content trains and grounds Gemini models outside Search. AI Overviews and AI Mode are built from the normal Googlebot index, so the only ways to limit them are the usual Search controls: noindex, nosnippet, data-nosnippet and max-snippet. Google's documentation states Google-Extended has no effect on Search inclusion or ranking.
Which AI crawlers should I block if I want to stay visible in ChatGPT and Claude?
Block the training crawlers and leave the search and user agents alone. That means disallowing GPTBot and ClaudeBot while allowing OAI-SearchBot, ChatGPT-User, Claude-SearchBot and Claude-User. OpenAI and Anthropic both document that each agent is controlled independently, so a training block does not affect whether their assistants can retrieve and cite your pages.
Does robots.txt actually stop AI bots?
Only the ones that choose to obey it. The major vendors say their crawlers honour robots.txt, but user-triggered fetchers from OpenAI, Perplexity and Meta are documented as able to bypass it, and Cloudflare caught Perplexity crawling blocked sites with undeclared user agents in 2025. If you need a hard block, enforce it at the CDN or firewall with verified IP lists, and treat robots.txt as a published preference.
Do I need an llms.txt file as well as robots.txt?
Not for visibility. Google has said no AI system uses llms.txt and that its own crawlers ignore it, and Google's AI features documentation says no extra files or markup are needed to appear in AI Overviews or AI Mode. Publishing one is harmless, and our generator will build it in seconds, but it is not a substitute for a correct robots.txt and a crawlable site.
