OpenAI, Google, Microsoft and Anthropic back shared AI scraping opt-out

OpenAI, Google, Microsoft and Anthropic say they will support a shared way for publishers to opt out of AI scraping, according to The Verge. The proposed control matters because website owners currently have to express their AI training preferences separately to each company.

The new signal is being proposed inside the Internet Engineering Task Force process by Kevin Kelley, a technical lead for standards at Anthropic. For publishers, creators and small businesses, the practical takeaway is simple: AI crawler policy may be moving from one-off blocks toward a common web standard.

A single signal inside robots.txt

The proposed mechanism is a shared “policy token” that would sit in a site’s robots.txt file. Robots.txt is already used by websites to indicate which crawlers can access a site for purposes such as search indexing, archiving or AI training.

Today, according to The Verge, sites must set AI training preferences separately for each AI company. The new token is intended to let publishers apply a preference across multiple AI providers at once.

That could reduce administrative work for media sites, blogs, documentation sites and small business websites that do not have large technical teams. Instead of tracking the separate crawler names and policy systems of every AI provider, site operators could have one place to express a broad preference — if the companies involved honor the signal.

Who says it will honor the opt-out

OpenAI, Google, Microsoft and Anthropic say they will restrict their AI crawlers from accessing a site when they detect the shared token, The Verge reports. Anthropic has already adopted the signal, while Google and Microsoft plan to add support next year. OpenAI says it will implement support in the “coming months.”

Meta communications head Andy Stone also indicated support for the idea, according to the report, but did not say whether Meta would follow the signal. Stone said a common preference signal for multiple providers is an important step, and that Meta looks forward to working with other AI companies on standards and implementation.

For users of products such as ChatGPT, Google Gemini and Claude, this is not a new app feature. It is infrastructure: a website-level control that could shape what AI companies’ crawlers are allowed to fetch for model-related uses.

The move also sits alongside broader tensions around AI systems and web content. We have seen related fights over publisher terms and scraping, including Meta’s disputed approach to publisher websites in recent policy coverage and its later delay after backlash over AI scraping terms /news/meta-delays-new-business-terms-after-publisher-backlash-over-ai-scraping.

Microsoft and Google point to more granular controls

Microsoft says it will provide more detailed options, including the ability for publishers to opt out of AI training or AI grounding. The Verge describes grounding as the process of directing an AI model to a verified information source to reduce hallucinations.

Microsoft also says websites will continue to appear in Bing search results regardless of those AI-related preferences. Google has already introduced similar controls, according to the report, allowing sites to decide whether their content appears in AI search features or is used for AI training without affecting search visibility.

That distinction is important for small publishers and companies that depend on search traffic. Many site owners may want visibility in traditional search while objecting to some AI training or AI answer uses. The direction described by Microsoft and Google suggests the debate is shifting from a simple allow-or-block choice toward separate controls for different crawler purposes.

What publishers should watch next

The announcement does not end the scraping debate. Other approaches are also emerging, including the Really Simple Licensing standard, which The Verge says gives publishers a way to charge crawlers that visit their sites. Cloudflare has also moved to block AI crawlers by default, according to the report.

For website owners, the immediate questions are practical:

• Which AI companies will actually implement the shared token?
• How will the token interact with existing robots.txt rules?
• Will Meta or other large AI companies commit to honoring it?
• How granular will the controls be across training, grounding, search and archiving?

For creators, marketers and small businesses, this is worth tracking even if you do not publish news. Product pages, blog posts, help centers and portfolios are all web content that crawlers may encounter. A shared opt-out signal could make AI data policy easier to manage, but only if adoption becomes broad and the instructions are clear enough for non-specialists to use safely.