- Cloudflare has replaced its single “block AI bots” switch with three behavioral categories: Search, Agent, and Training, available to all customers including free accounts, effective immediately.
- Starting September 15, multi-purpose crawlers like Googlebot, Applebot, and Bingbot will be blocked on any site where Training is blocked, because the strictest applicable rule now governs combined-purpose crawlers.
Cloudflare rolled out new AI traffic controls this week, extending the ability to manage bot behavior to every customer on its network, including those on the free tier. The update replaces a single “block AI bots” toggle with three separate categories: Search, Agent, and Training.
On the surface, that reads as a straightforward improvement for site owners who wanted more precision. For many, it will be. What the headline does not say is that a default change arriving September 15 could pull Googlebot, Applebot, and Bingbot off sites that believed they were only protecting themselves from AI training crawlers.
AI search consultant Alex Groberman warned that many businesses will hear “block AI bots” and treat it as a safe default, or they will hear “AI search visibility” and assume the settings should stay fully open. According to Groberman, both readings miss what actually matters: knowing which bots drive discovery, which drive action, and which simply extract content with nothing sent back.
Cloudflare just rolled out the most important AI search update of 2026.
If your site uses Cloudflare and you want to continue showing up in ChatGPT, Claude, Google’s AI family, etc – pay attention.
A lot of site owners are going to misunderstand what is happening and pay the… https://t.co/5ooszHXg4E pic.twitter.com/35VaJj9EC7
— Alex Groberman (@alexgroberman) July 2, 2026
What Each Category Actually Covers
The taxonomy Cloudflare introduced on July 1, 2026 is built around behavior, not identity. Search crawlers scan and index content so AI systems can pull answers from it later; Cloudflare connects this category to referral traffic, treating it as the behavior most site owners have a direct commercial interest in allowing.
Agent crawlers work differently: they operate in real time on a person’s behalf, visiting pages to complete a task someone assigned to a chatbot, AI assistant, or browser agent.
Training crawling carries the most friction for publishers. Its purpose is to absorb content into an AI model permanently, contributing to its underlying capabilities with no inherent mechanism to send traffic back to the source.
The Controls Are Live. September 15 Is When the Defaults Shift
All three management options are available right now to every Cloudflare customer. The change that carries real urgency lands on September 15, when Cloudflare updates default settings for two groups: new domains joining its network, and existing free-tier customers who have not manually adjusted their settings before that date.
For both groups, Training and Agent crawlers will be blocked by default on pages that carry ads, while Search crawlers will remain permitted. As Cloudflare’s product leads Jin-Hee Lee and Bryan Becker explained in the announcement, an ad on a page signals that the site owner built that page for human attention. Bots that bypass that equation, whether training scrapers or automated agents, work against the page’s commercial intent.
The Googlebot Problem That Is Not Getting Enough Attention
What makes the September 15 change genuinely disruptive, rather than a routine settings update, is the way Cloudflare is now treating multi-purpose crawlers. Several of the most consequential bots on the web do not operate inside a single category.
As Search Engine Journal’s Matt G. Southern reported, Googlebot, Applebot, and Bingbot each perform both search indexing and AI training. Under Cloudflare’s updated framework, a crawler that serves multiple purposes inherits every applicable policy, and the most restrictive one governs.
A site that has blocked Training will therefore also block any crawler that combines Search with Training. That includes Googlebot. It includes Applebot. It includes Bingbot. The block happens at the network level, not through a robots.txt directive that those crawlers could technically disregard.
Training Is Now the Majority of What Crawlers Do
This is not a theoretical edge case catching a small slice of traffic. Cloudflare’s agentic internet bot report, published alongside the announcement, shows AI training now accounts for 52 percent of all crawler requests on its network as of June 2026, rising sharply from 22 percent in spring 2025.
Mixed-use crawlers, those blending search indexing, agent behavior, and training functions into a single bot, make up over 36 percent of all crawler activity. Pure search crawling, the category that historically sent referral traffic back to publishers in exchange for content access, has become a small and shrinking portion of what is actually hitting most sites.
Who Is Most Exposed Before September 15?
The site owners with the most immediate risk from the September 15 change are those who switched on Cloudflare’s legacy “Block AI Bots” option when it launched in July 2025. At that point, the setting targeted single-purpose training crawlers, and the consequences were clear.
Under the new framework, that same legacy setting now activates the strictest-rule logic for multi-purpose crawlers as well, putting Googlebot in scope without the site owner having made any new decision. As Cloudflare’s press release confirms, anyone who turned on that block and has not revisited their dashboard since is directly affected.
Opting out requires going into Security settings and explicitly confirming that Training crawlers which also perform Search functions should not be changed. It takes minutes, but only if site owners know the deadline exists.
The Scale of the Shift Behind the Policy
The policy change sits inside a broader shift in web traffic. Cloudflare says more than 50 percent of internet traffic is now non-human, and that crawler behavior is moving away from simple search indexing toward training, agent activity, and mixed-use crawling.
That acceleration is the structural force that made a binary block switch an inadequate tool. Cloudflare’s network sits in front of more than 20 percent of all web domains, which gives the company a signal on crawler behavior that most infrastructure providers cannot replicate.
BotBase and Content-Use Signals for Enterprise Customers
Enterprise Bot Management customers get a new visibility layer through BotBase, a searchable directory of every tracked bot and AI agent Cloudflare has catalogued, each classified by behavior type. Administrators can filter traffic by individual bot, review its classification, and copy detection IDs for use directly inside security rules.
Cloudflare says BotBase will expand later in 2026 into a full control center for known automated content, but for now the emphasis is on visibility. Running alongside BotBase is a content-use signal system that extends Content Signals in robots.txt, letting site owners declare how they want bots to handle their content after crawling it.
The three levels move from most to least restrictive: immediate, meaning no storage or reuse; reference, which is now the default, covering indexing with excerpts and links back; and full, which permits summarization and reproduction.
What Does “Verified” Mean Now?
The Verified Bot designation has been redefined in a way that matters for how site owners read access logs. Previously, earning Verified status on Cloudflare’s network meant a bot was allowed by default. That is no longer the case.
Verified status now confirms identity and honest representation, nothing more. Whether a Verified bot can access a site depends entirely on how the site owner has configured their category-level policies for Search, Agent, and Training traffic.
Any bot that ignores a site’s declared content-use preference, or reproduces content in full against the site owner’s stated settings, loses its Verified status. Bots that reproduce content entirely are already ineligible for Verified classification under the updated rules.
The Configuration Question Differs by Business Type
There is no single correct setting here, which is part of what makes this update genuinely difficult to apply well.
Ad-supported publishers have the most straightforward case for blocking Training and Agent crawlers by default: those categories return no referral revenue and create real content-risk exposure with no upside.
SaaS companies are in a different position. The AI agents browsing product documentation, pricing pages, and head-to-head comparisons are often the exact systems surfacing that company to a buyer who is mid-decision. Cutting off agent access removes visibility at the most commercially valuable moment in the research process.
This is a tension we have covered in depth in our look at how agentic browsing is changing what it means for a site to be machine-readable: the permission layer and the content layer now both need to be right.
Ecommerce businesses land somewhere between the two: training scrapers harvesting product data without any compensation arrangement is a genuine problem, while AI search agents driving qualified buyers toward category or product pages represent a discovery channel that most ecommerce teams want active.
The practical takeaway is simple: site owners should check their Cloudflare AI bot settings before September 15, especially if they previously enabled the legacy “Block AI Bots” option. Blocking Training may be the right choice, but it should be a deliberate choice, not an accidental search visibility problem.
