OpenAI, Google and Anthropic back a new scraping opt-out
OpenAI, Google, Microsoft and Anthropic say they will honour one shared robots.txt signal that lets publishers opt out of AI training across all of them.
The facts
- According to The Verge, OpenAI, Google, Microsoft and Anthropic say they will adopt a shared "policy token" that lets publishers set AI training preferences once, rather than company by company.
- The signal was proposed by Kevin Kelley, a technical lead for standards at Anthropic, through the IETF, and it sits inside a site's robots.txt file.
- The four companies say they will restrict their AI crawlers from a site that carries the token. Microsoft will also let publishers opt out of AI training or AI grounding separately, and sites will still appear in Bing search results.
- Anthropic has already adopted the signal. Google and Microsoft plan to support it next year, and OpenAI says it will do so in the "coming months".
Case notes · Danny's take
Why it matters
Until now, a publisher who wanted out of AI training had to chase each company separately. One signal is easier to manage. But a signal is a request, not a lock. It only works if the crawlers keep their word, and the timelines are staggered: only Anthropic supports it today. The bigger shift is that the biggest AI firms are now on record agreeing that publishers should get a say.
For builders
If you run a site, blog or content business, watch for the policy token in robots.txt guidance and check your server logs to see which crawlers actually respect it. If you build crawlers or data pipelines, plan to honour the signal early rather than be forced to later.
Not financial advice. Do your own investigation.
This story was summarised with AI from The Verge's reporting and checked against our editorial policy. Danny's take reflects his views. Disclaimer.
Danny Walter
Founder of Moon Republic, Volt Terminal and Moon Republic Academy. Author of The Market Is a Crime Scene and After the Applause.