A common switch for a messy problem
The working headline is the story: four giants are trying to unify AI opt-outs. On October 30, 2026, Google, Microsoft and OpenAI published support for a shared signal in robots.txt that allows website operators to express whether their content may be used for generative AI training, while reporting on the same announcement identifies Anthropic as the fourth backer through Kevin Kelley, the Anthropic author of the underlying Internet-Draft . The result is not a new privacy dashboard for individual app users; it is a machine-readable preference for websites, aimed at the publishers, creators, brands and platforms that have spent years maintaining lists of AI crawlers one company at a time.
The practical problem is easy to understand. A site that wanted to avoid AI training previously had to know which crawler belonged to which company, what each company’s control meant, and whether blocking a crawler would also remove the site from search, citation or answer experiences. Google had Google-Extended, OpenAI had GPTBot, Anthropic had ClaudeBot, and Microsoft used its own publisher controls, including NoArchive and NoCache for AI-related uses . The shared proposal reduces that maintenance burden by letting participating providers read one common instruction.
The token: __usage_ai-training
The new mechanism is built around a policy token called __usage_ai-training, defined in the Internet-Draft “Machine-Readable Training Preference Signal,” last updated October 30, 2026 . A website can place an ordinary robots.txt group under that token and then use familiar Allow and Disallow rules to express a preference by site or path . In simple terms, User-Agent: __usage_ai-training followed by Disallow: / says the operator does not want the site used for the training purpose covered by the token.
That detail matters: the draft is not inventing a whole new syntax. It relies on the existing structure of robots.txt, including path matching and precedence, while adding a purpose-based label in the User-Agent line . It also says wildcard groups such as User-Agent: * must not be treated as matching the new policy token, and if the token is absent, no shared training preference has been expressed . That means today’s generic bot blocks do not automatically become AI-training opt-outs.
The draft also separates use from access. It states that the policy token is not itself a crawling directive and that crawler-specific rules continue to operate independently . In practice, that distinction is crucial: a site might allow crawling for search or live answer citation while objecting to model training. The shared token is meant to express the latter preference, not necessarily to stop a bot from fetching a URL.
Same signal, different company behavior
The four-company alignment is real, but it is not uniform implementation. Google says that when a website uses the shared token to opt out of training and has not explicitly specified Google-Extended, Google plans to interpret the shared opt-out as if the publisher had disallowed Google-Extended . Google also says existing company-specific controls will continue to work, and it aims to support the new control in 2027 . Its crawler documentation, cited in current reporting, narrows that target to early 2027 .
Microsoft’s approach is more granular. Microsoft says it is building on the shared AI Training preference while adding a separate AI Grounding preference, so publishers can distinguish between training future models and allowing content to inform generated responses in products such as Bing, Copilot and Microsoft 365 . Microsoft plans to begin honoring the new preferences as it crawls on February 1, 2027, and it says sites using NoArchive or NoCache must move to the new preferences by March 1, 2027 if they want to maintain restrictions . Microsoft also adds a page-level training option that can override the site-level robots.txt setting .
OpenAI’s announcement is shorter and more directional. It says OpenAI will honor the common signal in robots.txt, describes participation as voluntary, and says implementation is planned “in the coming months” . OpenAI also says its existing crawler controls will continue to apply and be respected . That is important for publishers because the shared signal is not yet live everywhere; until each provider actually implements it, existing vendor-specific rules remain operationally relevant.
Anthropic’s role is slightly different. The Internet-Draft is authored by Kevin Kelley and lists Anthropic in the author block . Reporting on the announcement says Anthropic has started implementing support, though without a firm completion date . That makes Anthropic both the source of the draft proposal and one of the participating providers, but the draft itself is still an individual Internet-Draft, not an adopted IETF standard .
Why this matters for consent and compliance
For publishers, the promise is administrative simplicity. A common signal lowers the cost of saying “no training” consistently across providers that support it. That can matter for large media groups with thousands of URLs, small creators without legal teams, and software platforms hosting user-generated content. Instead of treating every AI crawler as a new side quest, site operators get a single preference channel that can be reused as more providers join.
For AI companies, the shared signal offers a cleaner compliance story. Google says the proposal reflects feedback from publishers, civil society groups, regulators and others calling for industry-wide AI training controls, and it links the proposal to the EU General-Purpose AI Code of Practice under the EU AI Act, which recognizes robots.txt as a primary method for machine-readable publisher preferences . Even where the law does not force one exact technical format, an interoperable signal makes it easier to show that a provider had a process for receiving and honoring publisher choices.
For users and creators, the impact is indirect but meaningful. The mechanism does not give a person a universal switch across every platform where their work appears. It helps the operator of a domain express preferences for that domain. Still, if adopted widely, it could reduce the gap between having rights in theory and having a machine-readable way to communicate them at web scale.
The limits: voluntary, future-facing and not a magic eraser
The biggest caveat is that the proposal is voluntary. The IETF page explicitly says the draft is an Internet-Draft, that anyone may submit one, and that it has no formal standing in the IETF standards process . The document also has an expiry date of May 3, 2027 unless revised or replaced . In other words, the industry has a serious candidate for a common preference signal, not a completed global standard.
The second limit is enforcement. A robots.txt preference is only as strong as the systems that choose to read and honor it. The draft defines how a preference can be expressed, but it does not itself create audits, penalties or compensation . Microsoft is explicit that its choices do not guarantee inclusion, traffic or payment, and that restricting grounding can reduce visibility in AI experiences .
The third limit is timing. Google targets 2027, Microsoft starts February 1, 2027, OpenAI says the coming months, and Anthropic has begun implementation without a public end date . A publisher adding the token today may be making a forward-looking declaration, not flipping an immediately universal switch.
The fourth limit is previously ingested data. None of the current official announcements clearly says that the shared signal will remove content already collected or reverse training that has already occurred . The most reliable reading is therefore cautious: the token is designed to express preferences for future use as providers crawl or process content under their own implementation schedules.
What site operators should do now
The immediate recommendation is not to delete existing crawler rules. Google, Microsoft and OpenAI all frame the shared signal as coexisting with current controls, and Microsoft is the only one with a detailed replacement calendar for older AI preferences . Operators should add the shared token if they want to express the common preference, but keep vendor-specific controls until each provider’s implementation is live and documented.
They should also decide whether the issue is training, grounding, crawling or all three. Microsoft now separates training from grounding, while Google’s Google-Extended framework connects its training control to Gemini-related grounding eligibility . That difference means the same business goal may require different settings by provider.
The larger significance is that AI opt-outs are starting to look like infrastructure rather than ad hoc pleading. A universal API for privacy preferences is still more aspiration than reality, but the four-company alignment around __usage_ai-training is a concrete step toward a web where consent can be expressed once and interpreted consistently. Whether it becomes meaningful will depend on adoption, implementation detail, enforcement and whether the next wave of AI providers decides to join rather than route around it.
Comments
Be the first to comment.