OpenAI, Google, Microsoft and Anthropic have agreed to adopt a new standard that allows publishers to opt out of having their content scraped for AI training. This shared policy token signal is proposed by Kevin Kelley at Anthropic and will be managed through the Internet Engineering Task Force. The mechanism operates within a website’s robots.txt file, the same location that currently directs web crawlers regarding search indexing and archiving permissions.
The move aims to let content creators apply their training preferences across multiple major AI companies simultaneously. Previously, publishers had to configure these settings individually for each provider, creating a significant administrative burden. By centralising the signal in a standard file, the initiative reduces the friction of managing consent across the industry.
* The standard is currently being developed under the IETF framework.
* It replaces the need for separate agreements with every large model provider.
* Publishers can now enforce their scraping preferences through a single technical directive.
