Google Will Honor __usage_ai-training, a Shared robots.txt Line That Opts a Site Out of AI Training Across Participating Companies, Starting in 2027

The draft is short, and the trick is simple. Instead of listing GPTBot, Google-Extended, ClaudeBot and every other training crawler by name, a site adds one group:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

That example comes straight from the draft. It means “don’t train on anything here except the press kit.” The draft calls __usage_ai-training a policy token, and it “need not appear in the User-Agent header field” of any request. It’s a label for a purpose rather than the name of a bot.

The draft was published October 30 by Kevin Kelley of Anthropic. Its acknowledgments thank people from Google, Microsoft, OpenAI, Anthropic, the Internet Archive, News Corp and others for shaping the IETF AI Preferences working group’s discussion. “AI training” here has a specific meaning the working group agreed on: “Using an asset to modify the learned parameters of a generative AI model.”

The detail that will catch people is the wildcard rule. Lots of sites already have a User-agent: * group that blocks everything. Under this draft, “wildcard groups (i.e., User-Agent: *) MUST NOT be interpreted as matching the policy token.” And if there’s no __usage_ai-training group at all, “then no preference has been expressed.” A site that blocks every bot has, as far as this signal goes, said nothing about training.

The draft also keeps the old crawler rules separate. The token “is not a directive for crawling behavior,” so if you want a bot to stop fetching your pages, you still block that bot by name.

Here’s the part the announcement doesn’t put in the headline. Nothing changes at Google until 2027. The draft says “Private agreements MAY override preferences expressed via this mechanism,” and companies may treat their own crawler-specific rules as more specific than the shared one. So a publisher’s one-line opt-out can be outranked by a licensing deal or a vendor’s own token. That still beats the current mess, where opting out means chasing a list of user agents that grows every month. But a shared signal is only as strong as the number of companies that agree to read it. Google has said yes for next year. Watch who else signs on.