OpenAI, Google, Microsoft and Anthropic back one robots.txt token for AI training
A single policy token in robots.txt now expresses whether your content may be used to train generative models. Microsoft splits training from grounding and sets a March 2027 migration deadline for older directives.
What happened
Summary of reporting by The VergeFour of the largest AI developers have agreed to honour one shared control in robots.txt for saying whether a site's content may be used to train generative models. OpenAI and Google set out their positions in October 2026 (OpenAI, Google Search Central). Microsoft followed with a wider set of controls (Search Engine Land), and The Verge reports the four-party commitment.
The mechanism is deliberately plain. A publisher adds a group to robots.txt keyed on a policy token placed in the User-Agent line, __usage_ai-training, using ordinary Allow and Disallow paths. There are no new directive names and no new grammar, so the parsers already deployed across the internet keep reading crawl rules as before and skip the policy group. The token and its semantics come out of the IETF's AI Preferences working group.
The definitions do real work. Under the AIPREF vocabulary, AI Training means using an asset to modify the learned parameters of a generative AI model. Training of models used only for non-generative tasks falls outside it, and so does retrieval at inference time. Content can therefore be kept out of model training while still being fetched and cited when someone asks a question.
Microsoft goes past the shared signal. It separates training from grounding and adds a third state, DisallowExtended, which lets content ground Microsoft-owned experiences such as Copilot and Microsoft 365 while excluding grounding services Microsoft sells to third parties. Publishers relying on the older NoArchive and NoCache directives must move to the new settings by 1 March 2027, or Microsoft stops reading those directives as AI preferences.
The Azrty take
One robots.txt token now governs four AI labs, and the wildcard group in your file expresses nothing
The business so what is blunt. How model developers may use your content has stopped being a patchwork of per-vendor meta tags and become one declarative policy that a compliance decision can set in a single line. For a GCC bank, telco, retailer or media group with hundreds of domains and a long tail of PDFs, microsites and archived press pages, it is now a governance artefact rather than an SEO chore. It also carries a deadline: publishers leaning on NoArchive or NoCache to keep content out of Copilot or grounding results must configure the new settings by 1 March 2027.
The mechanics are worth reading before your team acts, starting with the IETF draft that defines the signal, proposed by Kevin Kelley of Anthropic and published on 30 October 2026. The shared signal is a policy token, __usage_ai-training, placed in a robots.txt User-Agent line and combined with ordinary Allow and Disallow paths. Its vocabulary comes from the IETF AIPREF working group vocabulary draft, where AI Training means using an asset to modify the learned parameters of a generative AI model. Two consequences follow: training of purely discriminative models falls outside it, and inference-time retrieval falls outside it. An explicit Disallow is a preference against, an explicit Allow is a preference in favour, and no policy token at all means no preference was expressed. Microsoft is the outlier in expressiveness. It separates training from grounding and adds a third state, DisallowExtended, which permits grounding in Copilot and Microsoft 365 while blocking the grounding services Microsoft resells to third parties.
Most teams will get this wrong in four specific ways. They will assume User-Agent: * covers the policy. It does not: the signal is matched on the policy token itself, exactly as RFC 9309 matches product tokens, so a robots.txt file that only has a wildcard group expresses no AI training preference whatsoever. They will read Disallow on the policy token as a crawl block. It is not a directive for crawling behaviour, so content keeps being fetched and only the downstream use changes. They will treat the signal as a legal right. It is not, because private agreements can override the preferences, which leaves contracts and copyright as the enforcement layer, exactly where UAE and Saudi media, publishing and content licensing terms already live. And they will assume the standard is finished. The core preference and vocabulary sections are not yet settled, and OpenAI has said it will expand and refine the mechanism. Whatever you write this quarter is version one of a policy that will move.
Our technical approach is policy-as-code on both sides of the fence. On the publishing side, we would hold one YAML matrix of the two axes you actually care about, training and grounding, plus the surface and provider carve-outs, and generate robots.txt along with the page-level meta and X-Robots-Tag headers from it in CI. Hand-editing robots.txt across a domain estate is how the wildcard footgun survives. The generated block keeps the policy group explicit and separate from crawl rules:
# Crawl rules: traffic management, unchanged
User-Agent: *
Disallow: /internal/
Usage policy: the policy group must be explicit. User-Agent: * does not match the policy token. User-Agent: __usage_ai-training Disallow: / Allow: /press-kit/
User-Agent: GPTBot Disallow: /
On the consumption side, the symmetry matters more than most engineering leaders expect. If you build retrieval pipelines and agents over public web content, there is now a cheap, standard way to honour other publishers' terms, and a growing expectation that you do. We would gate corpus ingestion on the policy token at the crawl job, store the allow, disallow or unknown result with each document for audit, and route model traffic through a gateway where budgets, provider choice and retention sit in one place. FastLLM Proxy is the natural control point for the model side, and Kuvryn AI for running those agents and models on your own GPU servers rather than shipping content out. Our team in the UAE does that deployment and the support behind it. The opportunity is real: a clear, defensible content usage policy is a negotiating asset when a lab or a licensing counterparty comes asking. The risk is equally real. A blanket Disallow: / on the training token, issued as a reflex from a board paper about protecting IP, will quietly remove your brand from the answer engines that increasingly mediate discovery across the Gulf, without buying you any protection that a signed agreement would not have bought better.
What to do now
- Add an explicit `User-Agent: __usage_ai-training` group to robots.txt on every production domain this quarter. Do not rely on `User-Agent: *`, which does not match the policy token. If you want training allowed on a press or media-kit path, express it with an `Allow:` line inside that policy group.
- Inventory every page carrying `noarchive` or `nocache`. If you want those restrictions preserved, set the new page-level AI Training and AI Grounding `meta` tags or `X-Robots-Tag` headers, then confirm in Bing Webmaster Tools before 1 March 2027.
- Decide the two axes separately for each content class: AI Training and AI Grounding. For owned thought leadership and product pages, consider allowing grounding with `DisallowExtended`, which keeps Copilot, Microsoft 365 and AI Overviews citations while blocking Microsoft's third-party grounding services.
- If you run your own RAG or agent ingestion over public web content, add a policy-token check to the crawl job next sprint: parse robots.txt, match `__usage_ai-training` as a product token, and store the allow, disallow or unknown decision with each document.
- Keep the file generated, not hand-edited. Hold the training and grounding decisions in a versioned config and render robots.txt in CI, so one policy change does not have to be repeated across hundreds of domains.