Cloudflare is making it harder for AI companies to scrape the open web for free. Starting September 15, 2026, the company’s default settings will block “mixed-use” crawlers, bots that both index content for search and collect it for AI training or agentic services, from any page that hosts advertisements.
The policy shift affects new Cloudflare customers, new sites set up by existing customers, and all existing free-tier customers by default. Site owners can opt in to allow mixed-use crawlers, but the burden has shifted: staying open to AI crawlers is now a choice, not the default.
Why now
Cloudflare co-founder and CEO Matthew Prince framed the move as a response to a structural shift in the internet. “Now that the majority of traffic on the internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge,” he said in a statement.
The company’s data shows that bots have surpassed human traffic online for the first time, a milestone that was not expected until 2027. Of that bot traffic, over 50 percent of AI crawler activity consists of re-fetching pages that have not changed, wasting publisher bandwidth and compute resources.
Google’s advantage
Cloudflare’s announcement takes direct aim at Google’s integrated crawler setup. Google provides a separate bot called Google Extended that lets publishers opt out of AI training and agent use without affecting their search visibility. But Cloudflare argues that Google’s default configuration gives it access to roughly twice as much data as competing AI companies, because publishers cannot easily block Google’s main crawler without also losing search placement.
“Google has access to 2x more information than other AI companies,” Prince noted, calling for mixed-use crawlers to separate search from agent use and training.
The pay-per-use model
The new policy builds on Cloudflare’s earlier efforts to give publishers leverage over AI companies. In 2024, the company launched tools to block unauthorized AI bots. In 2025, it introduced a marketplace where websites could charge for crawling, calling it “Pay Per Crawl.” The new policy evolves that into a “Pay Per Use” model, where publishers are compensated not just when content is fetched, but when it creates value in an AI product.
Initial partners include Ceramic.ai, which pays publishers when their content appears in its AI search results, and You.com, which pays for access to premium content.
What it means for AI companies
The September 15 deadline gives AI companies roughly two and a half months to separate their search indexing crawlers from their training and agent crawlers, or lose access to a significant portion of the ad-supported web. Companies that fail to comply will find their bots blocked by default on Cloudflare’s network, which serves roughly 20 percent of the web.
Sources: Cloudflare’s new policy pushes AI companies to pay (TechCrunch, July 1, 2026)