Training Preference Signal
also called AI training opt-out, Training opt-out token
One robots.txt line that tells every participating lab at once: do not train on this.
A machine-readable preference, written as an ordinary robots.txt group, stating that a site's content should not be used to train generative AI models. The shared token is __usage_ai-training: a site writes User-Agent: __usage_ai-training with Disallow rules, and every participating provider reads the same group instead of its own per-company crawler name. Google, Microsoft and OpenAI announced support on October 30, 2026, under an individual IETF Internet-Draft by Anthropic's Kevin Kelley.
The token is a usage preference, not a crawl rule. The draft states plainly that it is not a directive for crawling behaviour, so bots may still visit and cache a page, and search indexing and AI-written answers follow separate rules. A wildcard group (User-Agent: *) does not count: if no group names the token exactly, no preference has been expressed. Allow lines carve paths back in, as in the draft's example that opts a whole site out except /press-kit/.
Treat the October 2026 version as a promise with staggered delivery, not a switch you can flip today. Microsoft says Bingbot starts honouring the preferences on February 1, 2027; Google aims for 2027; OpenAI says the coming months. Anthropic's support is reported through the draft author, not confirmed by an Anthropic announcement of its own, and the draft itself is an individual submission with no formal IETF standing, expiring May 3, 2027 unless revised. It also reaches only future training: it does not remove content already collected or already trained into weights.
The mechanism adds no new robots.txt grammar. It reuses ordinary groups, path matching and longest-match precedence from RFC 9309, with the policy token in place of a crawler name. That design choice is why adoption could be announced by several labs at once: each provider only has to teach its crawler to recognise one more User-Agent value. It is also why the token is fragile in one specific way: a site that already blocks everything with User-Agent: * has expressed no training preference at all, because wildcard groups must not be interpreted as matching the token.
Opting out without vanishing from search
A publisher wants its articles out of training data but still indexed for search.
Add the __usage_ai-training group with Disallow: / and leave the search crawler groups alone. The training preference does not change search eligibility, and Google treats the token as mapping to Google-Extended, which it says is not a ranking signal. Check each provider's rollout date before promising readers the preference is live: at announcement, only the dates were staggered, with Microsoft firmest at February 1, 2027.
Treating the token as a block on crawling, search or AI answers, or as retroactive deletion. It is a forward-looking training preference, voluntarily honoured, and only by the providers that have committed to it.
- reportOpenAI to Honor Shared Robots.txt Signal for AI Training Preferences ↗
October 30, 2026 announcement coverage: the __usage_ai-training token, the individual Internet-Draft draft-kelley-aipref-training-preference-signal-00, and the point that an individual draft has no formal IETF standing.
- reportOne robots.txt rule to opt out of AI training: how it works ↗
Staggered rollout dates (Microsoft February 1, 2027; Google 2027; OpenAI in the coming months), the training-only scope, and the note that Anthropic is on board per reporting, with its own announcement not published.
Data Governance
Rules about what data may be used, by whom, and retained how long.
Privacy
Protecting the people whose data flows through the system.
Training
The expensive one-off process that sets the model’s parameters.
Pretraining
The first, most expensive pass: predict text at internet scale.
Guardrails
Constraints on what the system will do, enforced in code.