Cloudflare is enforcing a new rule that blocks mixed-use web crawlers from sites running ads by default. This change takes effect on September 15, 2026. The policy applies to new customers, new sites on existing accounts, and all current free users.
The block targets bots that combine search, agent work, and training in a single run. Site owners must adjust settings to allow access. The move aims to stop AI companies from scraping content without paying publishers or getting explicit permission.
Why the shift
Most website owners want their pages found by search engines and AI tools. They do not want their intellectual property given away for free. Cloudflare notes that the world’s largest search engine currently accesses roughly twice as much information as other AI firms. This happens because the search giant makes it hard for customers to remain discoverable without being used for AI training.
Google has argued against this claim. The company offers a bot called Google Extended. Site owners can opt out of having their content used for training or services like Gemini Apps and Vertex API. This option does not affect a site’s inclusion in Google Search. However, Googlebot still crawls for search, including features like AI Overviews and AI Mode.
“Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge,” said Matthew Prince. He is the co-founder and CEO of Cloudflare. Prince made this statement when announcing the news. He referred to the recent milestone where bots surpassed human traffic online for the first time. That shift was not expected to occur until next year.
“Cloudflare’s new tools and partnerships give website owners increased visibility and commercial opportunities and benefit AI companies that have bots with clear and transparent intent,” Prince said. “We hope that our proposed default changes encourage mixed-use crawlers to separate out search from agent use and training.”
Cloudflare sells products to help users launch their own AI systems. The company has also released tools to give publishers more control over their content. In recent years, Cloudflare launched tools to combat AI bots. This included a marketplace that lets websites charge AI bots for scraping, known as Pay Per Crawl.
The latter is now evolving into Pay Per Use. This allows publishers to charge AI companies when their content creates value, not just when it is fetched.
The change could also help conserve publishers’ bandwidth and compute resources for AI model providers. Cloudflare’s data suggested that over 50% of crawl traffic from AI crawlers is spent re-fetching unchanged pages.
Partnerships and payment
Cloudflare is initially working with two partners, Ceramic.ai and You.com. When a publisher opts in, they are paid when their content appears in Ceramic’s AI search results or when You.com accesses a piece of their premium content.
Other AI companies can customise this model for how they work, Cloudflare says.
What it means
Developers building AI agents or training models will face stricter access rules. They can no longer rely on default settings to scrape web pages that host advertisements. To access that data, companies must negotiate directly with site owners or implement a payment model. Publishers gain a mechanism to monetise their content based on its use by AI systems rather than just its retrieval.
