AI Training Opt-Out Robots.txt Token Explained: One Line to Block OpenAI, Google, Microsoft and Anthropic

Image: Wikimedia Commons / BalticServers, CC BY-SA 3.0 (illustrative image — the token is a text setting, not a device).

TL;DR: The AI training opt out robots.txt token explained in one line: OpenAI, Google, Microsoft and Anthropic will honour a single shared robots.txt entry — User-Agent: __usage_ai-training with Disallow: / — that tells all four labs not to use a site's content for AI training. Anthropic has started already; OpenAI follows in the coming months and Google and Microsoft in 2027. It covers future training only, not search results, AI answers, or content already collected.

📌 KEY HIGHLIGHTS

• One shared policy token, __usage_ai-training, replaces separate opt-outs for each AI lab's crawlers.
• Announced 30 October 2026, based on an IETF Internet-Draft by Anthropic's Kevin Kelley — a draft, not yet a formal standard.
• Anthropic has begun implementing it; OpenAI says "in the coming months"; Microsoft starts 1 February 2027; Google aims for 2027.
• It stops training use only — crawlers may still visit and index pages, and it does not remove content already used.
• Meta has signalled support but has not committed to honouring the token.

What happened?

On 30 October 2026, the four largest Western AI labs — OpenAI, Google, Microsoft and Anthropic — said they will honour one shared robots.txt signal that lets any website opt out of AI model training across all of them at once. The Verge's Emma Roth first reported the coordinated move, and Search Engine Roundtable and Search Engine Land covered the company announcements the same day.

Until now, a publisher that did not want its pages used to train AI models had to maintain a separate robots.txt rule for every company's crawlers: GPTBot for OpenAI, Google-Extended for Google, ClaudeBot for Anthropic, and Microsoft's equivalents, with new AI crawlers appearing all the time. The new signal collapses that chore into a single entry that every participating company agrees to read the same way.

The mechanism comes from the Internet Engineering Task Force's AI Preferences (AIPREF) working group. The specification is an individual Internet-Draft titled "Machine-Readable Training Preference Signal" (draft-kelley-aipref-training-preference-signal-00), authored by Kevin Kelley, Anthropic's technical lead for standards, dated 30 October 2026 and expiring 3 May 2027. Working-group participants include Meta, Cloudflare, the Common Crawl Foundation, News Corp, the Financial Times, the BBC, the Electronic Frontier Foundation and Creative Commons.

How the AI training opt out robots.txt token works

The token adds no new robots.txt grammar. A site owner writes an ordinary robots.txt group with the policy token in the User-Agent line. The draft's own example opts a whole site out of training while leaving its press kit open:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Three details in the draft matter for anyone writing the rule. First, the token must match exactly (apart from letter case). Second, a wildcard User-Agent: * group does not count — if no group names the token, the site has expressed no training preference at all. Third, company-specific tokens still take precedence: a site can set the shared token broadly and then fine-tune one provider's rule, such as Google-Extended, on top of it.

Google confirmed that behaviour: if a site uses the shared token to opt out and does not explicitly mention Google-Extended, Google plans to treat that as if Google-Extended had been disallowed. Sites that explicitly allow Google-Extended stay eligible for Gemini Apps grounding and the clicks that come with it. Google-Extended, its training control since 2023, is not going away.

Who honours it, and when

The rollout is staggered, not simultaneous — and at most of the four companies, nothing changes yet. The table below is the announced position as of 31 October 2026.

Company	Status	Timing
Anthropic	Implementing	Already started; rollout across services takes time
OpenAI	Committed	"In the coming months"; existing GPTBot controls keep working
Microsoft	Committed	Crawlers start honouring preferences from 1 February 2027, as pages are recrawled
Google	Committed	Aims to support the token in 2027, alongside other providers
Meta	Not committed	Communications head Andy Stone called a common signal "an important step"; no commitment to honour it

Microsoft goes furthest beyond the shared token. It is adding a page-level training setting and a separate "AI Grounding" setting that decides whether a page can feed live answers in Bing, Copilot and Microsoft 365, plus a DisallowExtended option that allows grounding in Microsoft's own products but not in the grounding services it sells to other companies. Microsoft's new preferences replace its older NoArchive and NoCache tags; sites relying on those tags have until 1 March 2027 to switch. Microsoft also says a site's preferences will not affect its eligibility in Bing search results.

Image: Wikimedia Commons / Virginia Tech, CC BY-SA 2.0 (illustrative image — AI training runs on servers like these).

What it does NOT do — the limits, honestly stated

The announcement is narrower than the headlines suggest, and the limits are stated in the draft and the company posts themselves:

• It is not a crawl block. The draft says plainly: "This policy token is not a directive for crawling behavior." Bots may still fetch, index and cache a page; the token governs only what happens to the copy afterwards, for training.
• It does not cover AI answers. Opting out of training will not keep a page out of AI-written search answers, AI Overviews or AI Mode — the use publishers blame for falling traffic. Reddit's CEO said in October 2026 that Google's AI search experience sent the platform over a million fewer weekly referrals during the month. That traffic problem needs different controls (such as nosnippet or data-nosnippet), not this token.
• It is not retroactive. Updating robots.txt does not remove material already collected into datasets or already trained into model weights. It blocks future training runs only.
• It is voluntary. The draft "is not endorsed by the IETF and has no formal standing," and OpenAI frames the mechanism as "a coordination mechanism rather than a security control" under RFC 9309. Non-compliant scrapers and smaller labs can simply ignore it.
• It is not a payment mechanism. The token lets publishers refuse training; it does not let them bill for it. That is a separate idea, covered by licensing schemes such as the Really Simple Licensing (RSL) standard.

The proposal is consistent with the EU Code of Practice for General Purpose AI model providers under the EU AI Act, which recognises robots.txt as a primary way for publishers to state machine-readable preferences. That regulatory tailwind is one reason four rivals moved together — but a code of practice is not the same as an enforceable rule.

Why it matters for India readers

No India-specific adoption has been announced, and no Indian law changes with this announcement — the India angle here is a sector signal, not a local event. But the same two lines will work for Indian news sites, blogs, creators and businesses as support lands, and Indian publishers face exactly the same scraping pressure as their Western counterparts: their reporting is crawled, summarised and answered by AI systems, often without a visit or a click back.

For a small Indian publisher, the practical point is sequencing. The company-specific rules that work today — Google-Extended, GPTBot and the rest — remain what actually works today, so they should stay in place. The shared token is worth adding now so coverage arrives automatically as each lab switches it on, but it does nothing yet at most of these companies. And any publisher weighing an opt-out should weigh the trade-off honestly: refusing training does not cost search visibility, but it may also mean their content never shapes the models their own readers use.

Should you care?

✅ Verdict: If you run a website, yes — add the token alongside your existing rules and keep the existing rules until each lab's support is live. If you only read the web, the change is invisible for now: it is a publishers' tool, not a consumer feature, and its real test comes in 2027 when Google and Microsoft switch it on and the IETF working group decides whether the draft becomes a standard.

FAQs

How do I opt out of AI training in robots.txt? Add a group naming the shared policy token: User-Agent: __usage_ai-training followed by Disallow: /, with Allow: lines for any folders you want to keep open. Keep your existing company-specific rules (Google-Extended, GPTBot) in place until each lab's support for the shared token is live.

Does robots.txt stop AI companies from using my content? Only for training, only at companies that honour the token, and only for future training runs. It does not stop crawling or indexing, does not keep content out of AI search answers, and does not remove content already collected or trained on.

Does the AI training opt-out remove content already used for training? No. Nothing in the draft or the company announcements offers retroactive removal. The token expresses a preference for future use; models already trained on a site's content are unaffected.

Will opting out of AI training hurt my Google search ranking? No. Google says Google-Extended is not a ranking signal, and Microsoft says training preferences do not affect Bing search eligibility. The token separates training permission from search visibility by design — but it also does not protect a site from AI Overviews or AI Mode, which need different controls.

Takeaway: Four rival AI labs agreeing on one opt-out line is genuinely new — but the line is a draft, voluntary, training-only and mostly not live yet. Add it, keep your old rules, and judge it in 2027.

Follow World Tech on TechByCountry for tomorrow's update.

📸 Get tech in the last 24 hours on Instagram — follow @_techinlast24hrs for daily posts (9 AM–9 PM) and reels at 12:30 PM & 8:30 PM.

Sources

• IETF Internet-Draft: Machine-Readable Training Preference Signal (draft-kelley-aipref-training-preference-signal-00, Kevin Kelley, 30 Oct 2026)
• Skimwire: One robots.txt rule to opt out of AI training — how it works (30 Oct 2026)
• RR Tech Pulse: Opt out of AI training in robots.txt (30 Oct 2026)
• Silicon Report: OpenAI, Google, Microsoft, Anthropic back AI opt-out token (30 Oct 2026)
• The Verge (Emma Roth), 30 Oct 2026 — first report of the coordinated announcement; Search Engine Roundtable / Search Engine Land coverage of the company posts (cited via the sources above).

Related reading on TechByCountry: AssemblyAI TTS-1 text-to-speech voice agents explained; OpenAI AI agent monitoring system explained; OpenAI Model Spec update — five values explained.