Cloudflare blocks mixed bots by default on ad pages from September 15

From September 15, 2026, Cloudflare's default settings block so-called mixed bots on every page that displays ads. These are bots a single company uses both for classic search and for AI work. The company announced the change months in advance. Its reason: the old split between search crawlers and training crawlers no longer reads clearly to site owners.

Since most internet traffic no longer comes from people, we have to go further and move faster so that a durable ecosystem can emerge, said Cloudflare co-founder and CEO Matthew Prince.

The mechanism rests on separating intent. Cohen told Press Gazette that when a company runs separate bots for training, for search and for agentic actions, a site owner can block or admit each one on its own. The two-month gap between the announcement and the rollout was meant to give people time to adjust.

How much of that traffic there is

The scale data shows why the topic came back. A 51Degrees analysis looked at more than 3 billion visits to sites in the twelve months to May 2026. It found that the share of sessions identified as AI harvesters rose by 62.5 percent, from 8 to 13 percent of all sessions. AmazonBot was the busiest bot with 4.1 percent of visits. Meta-ExternalAgent accounted for 2.9 percent, BingBot for 1.35 percent, Anthropic for 0.19 percent, OpenAI for 0.09 percent, Perplexity for 0.04 percent and Google Gemini for 0.0003 percent of sessions. Cloudflare said earlier that agentic bots generate around 57.4 percent of global traffic.

The author of one analysis points out that the numbers do not add up to the whole. Google uses the same bot to collect content for AI as it does for search, so it was left out of the breakdown. That case, one bot with two uses, is the crux of the dispute. It is also why the block is harder than it looks.

The reach is large. Cloudflare handles around 20 percent of the world's sites at the network level. The Financial Times, The Atlantic, Ziff Davis, Conde Nast and the Associated Press already use the new tools. A pay-per-crawl mechanism, a fee for the mere entry of a bot, is being tested in a closed beta. Publishers report that the block speeds up licensing talks. Lack of access turned out to be a stronger argument than appeals for voluntary payment.

The cost to publishers is twofold. Harvester traffic consumes bandwidth, and AI answers take away visits that could have earned on ads. Hence the parallel growth of billing mechanisms and visibility dashboards. A block with no room to negotiate would quickly amount to invisibility.