Four AI labs back one robots.txt training opt-out, with dates spread into 2027

A publisher can state one training preference instead of keeping a separate rule for GPTBot, Google-Extended and ClaudeBot. The token creates no licence and no payment. Microsoft says so directly.

One line, no new grammar

The token comes from an Internet-Draft by Kevin Kelley, a standards lead at Anthropic. He submitted it to the IETF's AI Preferences working group on 30 October 2026.

A site writes User-Agent: __usage_ai-training, then Disallow: / and any Allow: lines it wants. The draft's own example blocks everything except a /press-kit/ folder. The draft adds no new directive names or grammar. A parser that has never heard of the token skips the group. A wildcard group, User-Agent: *, cannot match the token. A file with no such group states no preference at all.

Four companies, four dates

Microsoft will honour the shared preference from 1 February 2027, as Bing recrawls pages. Google aims for 2027, and its crawler documentation says early 2027. OpenAI says it aims to implement the signal in the coming months.

Anthropic has started. Kelley wrote that support across its services will take time. He expects inevitable complications and edge cases.

Google has described one specific reading. If a site opts out with the shared line and does not mention Google-Extended, Google plans to treat that as a Google-Extended block. Google-Extended is the training control Google has offered since 2023. John Mueller, its search advocate, wrote that the token will help websites express their AI training preferences across multiple AI providers in a single step.

A preference, not a block

The draft says the token is not a directive for crawling behaviour. It says private agreements may override preferences expressed through it. It covers content used to modify the learned parameters of a generative AI model. Models that only classify or rank fall outside it.

Microsoft has split training from grounding, the step where an AI answer pulls in a live page. Its page-level AI Grounding setting includes a DisallowExtended option. That limits use to Microsoft's own products, such as Copilot and Microsoft 365, and excludes the grounding services it sells to other companies. Microsoft warns that blocking grounding will significantly reduce a brand's visibility in AI-generated experiences. It does not change a page's place in Bing search results.

Why publishers are not satisfied

The Digital News Publishers Association is pursuing a case against OpenAI. The case is in India's Delhi High Court. It alleges that copyrighted Indian journalism was scraped to train ChatGPT, including from sites that had used robots.txt to keep AI crawlers away.

A forward-looking preference does not touch that.

The draft's standing is thin. The IETF datatracker says the IETF has not endorsed it and it has no formal standing in the standards process. It expires on 3 May 2027. OpenAI called it a coordination mechanism rather than a security control under RFC 9309. A scraper that ignores robots.txt will ignore this too.

What happens next

Sites can record a temporary grounding preference in Bing Webmaster Tools. The option starts on 15 December 2026. Microsoft's NoArchive and NoCache tags, in use since 2023, stop counting as a training or grounding preference after 1 March 2027.

Meta communications head Andy Stone voiced support for a common signal. Meta has not said whether its crawlers will honour it. Google says the approach is consistent with the EU Code of Practice for general-purpose AI under the AI Act. That code treats robots.txt as a main way for publishers to state machine-readable preferences.