Cloudflare is replacing its single-click block for all AI crawlers with a system that lets site owners manage access for three specific categories. Starting 15 September 2026, bots used for training models and acting as user agents will be blocked by default on pages containing advertisements, while standard search crawlers remain allowed. This change applies to customers on the free plan as well as enterprise accounts. The new rules force operators to split their crawlers by purpose, addressing a concern raised by CEO Matthew Prince regarding Google bundling its search and AI tools. Prince noted that bot traffic surpassed human traffic in June 2026, a milestone he did not expect until late 2027. Enterprise users gain access to BotBase, a dashboard database showing how each bot classifies and uses content. Verified bots will no longer receive automatic access; operators must now prove honest identification and adherence to rules based on their category.
The shift moves away from treating all automated traffic as hostile, allowing legitimate indexing while protecting revenue streams. It also changes the definition of trust, requiring proof of identity rather than just a label. The approach aims to clarify bot behaviour without stopping necessary data collection.
- Training and agent bots default to blocked on ad-supported pages
- Search crawlers retain full access regardless of ad presence
- Verified status requires proof of honest identification
