On July 1, Cloudflare announced that from September 15, 2026, it will block "mixed-use" AI crawlers by default on any page that carries ads. New domains, new sites from existing customers, and every free-tier account are in scope. If you run a business site behind Cloudflare, this default could change what AI systems are allowed to read on your pages without you touching a single setting.
Most of the coverage has framed this as a win for publishers and a headache for OpenAI and Google. That is one story. The story that matters for a normal business is simpler and more urgent: is your site about to disappear from AI answers, and did you even choose that?
What Cloudflare Actually Changed
Cloudflare is splitting AI crawlers into three buckets and treating them differently (Cloudflare blog):
- Search: bots that index pages so they can be ranked and linked. Allowed by default.
- Training: bots that scrape content to train AI models. Blocked by default on ad-supported pages.
- Agents: bots that fetch pages in real time to answer a user's question. Also blocked by default on ad-supported pages.
The friction point is bots that do all three under one name. Cloudflare calls these mixed-use crawlers, and it is giving AI companies until September 15 to separate them or accept being blocked by default on many sites (TechCrunch). Alongside this, its Pay Per Crawl marketplace is evolving into Pay Per Use, so publishers can charge AI companies when their content creates value rather than only when it is fetched.
The headline reads like a blanket block. It is not. Search stays open. That distinction is the whole ballgame.
The Part Everyone Is Getting Wrong
Blocking AI training crawlers does not remove you from Google's AI Overviews or AI Mode.
This trips people up constantly, so let me be plain about it. Google's AI Overviews and AI Mode are built on the Google Search index, which is fed by Googlebot, not by the training crawler. Google's own documentation states that blocking Google-Extended, its AI training token, has no effect on search rankings, indexation, or AI Overview eligibility (Google Search Central). You can say no to training and still show up in the AI answer.
So when Cloudflare blocks "training" by default, your presence in Google's AI features is not what is at risk. What you are opting out of is your content being used to train future models. For most businesses that is a fine trade, arguably a good one.
The category to watch is Agents. When ChatGPT or Perplexity fetches live pages to build an answer, that is agent traffic. Block it and you can drop out of the real-time answers those tools generate. That is the setting that touches your visibility, and it is the one worth being deliberate about.
Our Take: Don't Reflexively Block
Here is where we part ways with a lot of the commentary. The "make AI pay for your content" framing is built for large publishers with traffic, leverage, and a licensing team. That is not most of the businesses we work with in Cyprus and the USA.
If you are a law firm, a clinic, a SaaS startup, or a local service business, your problem is almost never that AI companies are getting rich off your blog. Your problem is that not enough people find you in the first place. Blocking the crawlers that put you inside AI answers, to protect content nobody is licensing anyway, is solving a problem you do not have while creating one you do.
We see this constantly. A business hears "AI is stealing content," flips on an aggressive block, and quietly removes itself from the exact surfaces where buyers are now asking questions. The upside was theoretical. The lost visibility is real.
The Pay Per Use marketplace is genuinely interesting, and we expect it to matter for media brands. But charging for crawls only works when someone actually wants to pay for your content. For the vast majority of business sites, the value is in being cited and visited, not in a per-crawl fee that will round to nothing.
What to Actually Do This Month
This is not a reason to panic. It is a reason to check your settings on purpose instead of inheriting a default you never chose.
- Find out if you are on Cloudflare and what tier. The new defaults hit new domains, new sites, and free accounts hardest. If you launched recently, assume the defaults now apply to you.
- Decide per bucket, not in one switch. Blocking training is usually fine. Blocking agents is the decision that affects whether ChatGPT and Perplexity can read you live. Treat them separately.
- Keep Search open. This should be obvious, but confirm it. Nothing good comes from blocking the crawlers that index you for ranking.
- Check your robots.txt against your Cloudflare rules. They can quietly contradict each other. If you want a clean, deliberate crawler policy, our guide on AI crawlers and robots.txt walks through which bots to allow and block and why.
- Match the policy to your actual business. If you license content, explore Pay Per Use. If you sell services and need to be found, stay open to agents.
The Bigger Shift Underneath This
Cloudflare's move is a signal, not a one-off. The open web is being carved into "who gets to read this, and on what terms." Search crawling, AI training, and AI agents used to travel together under one bot. Now they are being pulled apart, and every site owner is quietly being handed a set of switches most have never looked at.
For businesses chasing visibility in AI search, the winning position for the next while is boring but correct: stay readable to the systems that answer questions, say no to training if you want to, and stop treating "block AI" as a free move. It is not free. It has a cost measured in the customers who never find you.
If you are not sure what your current setup is telling AI crawlers, that is exactly the kind of thing a technical review surfaces fast. Our AI search optimization and technical SEO work starts by checking what you are actually letting the machines see, and you can get a read on your own position with a free SEO review.
The businesses that win in AI search over the next year will not be the ones who blocked the hardest. They will be the ones who understood which door they were closing before they closed it.
