🔍 Read the full analysis: Does The AI Training Opt-Out Cover All Your Data? on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
OpenAI, Google, Microsoft and Anthropic say they will support a shared robots.txt signal asking them not to use a website’s content to train generative AI models. The signal is voluntary, begins on different schedules, and does not block crawling or cover AI answers based on live web content; Meta has voiced support without committing to honor it.
OpenAI, Google, Microsoft and Anthropic said on October 30 that they will recognize a shared robots.txt signal asking them not to use a website’s content for generative AI training. The move could let publishers express that preference with one rule instead of separate instructions for each company, but it is a voluntary request—not a technical block—and does not cover every use of web content in AI products.
The shared instruction is a robots.txt rule group using User-Agent: __usage_ai-training and Disallow: /. It appears in an individual Internet-Draft titled “Machine-Readable Training Preference Signal,” published on October 30 by Kevin Kelley of Anthropic. The draft is associated with the Internet Engineering Task Force’s AI Preferences working group, but it is not an adopted internet standard.
Companies have set different timetables. Anthropic says it has already started implementation, while warning that support across its services will take time. OpenAI says it aims to implement the signal “in the coming months.” Google aims to support it in 2027; its crawler documentation specifies “early 2027.” Microsoft plans to honor it from February 1, 2027, as it crawls websites.
Microsoft also announced a separate page-level preference for “AI Grounding,” covering web content used to inform answers in Bing, Copilot and Microsoft 365. Microsoft says both preferences take effect on February 1, 2027. Its existing NoArchive and NoCache controls will stop counting as preferences for training or grounding after March 1, 2027. Google says the shared signal will be treated like a Google-Extended disallow if a site has no separate Google-Extended rule.
Infographic 01 · Who backs it, and when
One line in robots.txt, four companies, four timelines.
A shared signal against AI training. Only Anthropic says it has started; the others name dates or months. The standard itself is still a draft.
- 30 Oct 2026Draft published, four companies announce supportdoneIndividual Internet-Draft by Kevin Kelley (Anthropic)
- 30 Oct 2026Anthropic: “has already started implementing support”startedIts standards lead on the IETF list: “It’ll take us some time to add support across our services.”
- Coming monthsOpenAI aims to implement itplanExisting GPTBot and OAI-SearchBot rules keep applying
- 16 + 19 Nov 2026Working group meets at IETF 127, San FranciscoopenOn 21 Oct a chair said no consensus has been declared yet
- 1 Feb 2027Microsoft honors the new preferences as it crawlsplanTraining in robots.txt or per page; grounding per page
- 1 Mar 2027Bing stops reading NoArchive / NoCache as AI preferencesdeadlinePublish the new settings before this date
- 2027Google plans support; its docs say “early 2027”planA “no” maps to Google-Extended unless you set that token yourself
- 3 May 2027Draft version 00 expires unless updatedInternet-Drafts are valid for at most six months
Binding? No, voluntary
- OpenAI calls it a “voluntary effort”.
- The vocabulary draft does not “ensure that preferences are followed”.
- The draft “has no formal standing in the IETF standards process”.
- Private agreements may override it, the draft says.
Who else?
- Meta supports a common signal, but did not say whether it will honor it.
- Apple, Amazon, xAI, Perplexity: no statement in the sources reviewed for this article.
- The draft’s acknowledgments list contributors “for identification only”, not supporters.
Sources: IETF draft, 30 Oct 2026 · Anthropic on the IETF list · AIPREF minutes, 21 Oct 2026 · Google, 30 Oct 2026 · Google crawler docs · OpenAI, 30 Oct 2026 · Microsoft Bing, Oct 2026 · Meta (Andy Stone on X). Company plans are company statements.
Powered by Thorsten Meyer AI
One Rule, Several Different Uses
For publishers, a shared instruction could reduce the work of maintaining separate training opt-outs for each participating AI company. That matters because a site-specific rule only communicates a preference to services that recognize it. Until each company implements the shared signal, publishers may still need existing crawler controls to express their wishes to individual providers.
The distinction between training and AI answers is important. A site may want to prevent its material from being used to develop models while still allowing search indexing or use in responses grounded in current web pages. The shared rule addresses training, not all downstream uses. Microsoft’s separate grounding control makes that difference explicit; the common training signal alone does not tell a company to exclude a site from AI answers.
The policy also does not promise payment, traffic, or visibility. Microsoft said neither choice guarantees “inclusion, traffic, or payment.” The new mechanism gives publishers a way to state a preference, but the announcements do not establish a compensation system or guarantee how companies will respond to a site’s content in their products.
Products Worth Considering
Samsung 27 Odyssey OLED G5 (G50SF) QHD QD-OLED Gaming Monitor
Samsung 27-Inch Odyssey QD-OLED G8 4K 240Hz Gaming Monitor
LG 34GX90SA-W 34-inch Ultragear WQHD OLED Curved Gaming Monitor
As an affiliate, we earn on qualifying purchases.
Existing Crawler Rules Remain Relevant
Before this announcement, website operators could use different instructions for different crawlers, including GPTBot for OpenAI, Google-Extended and ClaudeBot. Microsoft also offered page-level NoArchive and NoCache tags. The shared signal is intended to provide a common expression of a training preference, rather than immediately replacing every company-specific setting.
According to the company announcements summarized by ThorstenMeyerAI.com, OpenAI says its existing crawler controls will continue to apply. Microsoft says its preferences do not change eligibility for Bing search results, while Google says Google-Extended does not affect Google Search. These are separate product and data-use choices: publishers may need to decide independently about model training, AI answer grounding and ordinary search.
The draft’s status is also limited. The source reports that the proposal has no formal standing in the IETF standards process. At an October 21 working-group meeting, a chair said the group had not declared consensus on anything. Company support for the draft is therefore not the same as approval of a formal standard.
““has already started implementing support for this draft””
— Anthropic
Voluntary Signal, Uneven Adoption
The rule is a request to participating companies, not a mechanism that technically prevents access to a site or guarantees compliance. It does not block crawling, remove material from models that have already been trained, or reach companies that do not adopt it. The source says it found no adoption statements from Apple, Amazon, xAI or Perplexity, and the list of future participants remains open.
Meta’s position is also unresolved: its spokesperson supported publishers’ ability to express preferences but did not commit to honoring this specific signal. Anthropic’s crawler help page still described its existing per-crawler controls as of October 31, according to the source. It is not yet clear how quickly each company will complete implementation across its services or how consistently the signal will be applied.
There is also a practical gap between stated preferences and outcomes. The announcements do not specify a shared enforcement process, audit mechanism or remedy if a company ignores a site’s instruction. Publishers should not treat the draft as proof that content has been excluded from training or AI-generated answers.
Watch Rollouts Through 2027
The next milestones are company implementation updates. Anthropic says support is underway, OpenAI targets the coming months, and Google aims for 2027, with its documentation saying early 2027. Microsoft’s stated start date is February 1, 2027; sites using NoArchive or NoCache for training or grounding preferences will need to account for the announced March 1 cutoff.
Website operators can review their robots.txt rules and decide separately whether to restrict training, AI grounding, or search-related access. The source advises keeping company-specific rules until each provider has switched to the common signal. Any change should be checked against the relevant provider’s current documentation, since rollout timing and product controls may change.
For the broader proposal, the next question is whether the IETF working group reaches consensus and whether more AI companies publicly commit to honoring the signal. Until then, the announcement is a coordinated step by four companies, not a universal rule for AI use of web content.
Key Questions
What does the shared AI training opt-out do?
It is a robots.txt preference asking participating companies not to use a website’s content to train generative AI models. It is voluntary and does not technically block access.
Does the signal stop AI answers from using my site?
Not by itself. The shared signal covers training, not AI answers grounded in current web content. Microsoft announced a separate AI Grounding preference for certain Microsoft products.
When will the companies honor the rule?
Timing varies: Anthropic says implementation has started, OpenAI targets the coming months, Google aims for 2027, and Microsoft says February 1, 2027. The companies have not described identical rollout schedules.
Has Meta agreed to honor the signal?
Meta’s spokesperson expressed support for publishers’ ability to state preferences, but did not commit to honoring this specific signal, according to the source.
Is the shared rule an official internet standard?
No. It is an individual Internet-Draft linked to the IETF’s AI Preferences work. The source says it has no formal standing in the IETF standards process and reports that the working group had not declared consensus.
Source: ThorstenMeyerAI.com
