AI and robots.txt: new controls and Google-Extended
Google plans to support a shared signal in robots.txt from early 2027 so website owners can indicate whether their content can be used to train generative AI models.
In the meantime, Google-Extended is already available. Restricting it has no effect on indexing or rankings in Google Search.
It can affect whether pages are used as sources in Gemini responses.
That distinction matters to a publisher that wants its reporting to stay visible in Google Search while limiting its use for AI training.
The right setting depends on where the publisher wants its reporting to appear and how it wants that material to be used.
What is Google-Extended and what does it control?
Google-Extended is an identifier that Google reads in the robots.txt file.
It goes on the User-agent line and does not represent a separate crawler.
Google still uses its existing crawlers, so you won’t see a bot called Google-Extended in your server logs.
The control covers two uses of content Google has already crawled.
- Training future generations of Gemini models that power Gemini Apps and the Gemini API in Vertex AI.
- Grounding in Gemini Apps and in Grounding with Google Search on Vertex AI. When the system receives a request, it can draw on Google’s index and use a page as a source for its answer without training the model at that point.
A publisher could, for example, exclude its /investigations/ section with a rule for Google-Extended while still letting Googlebot crawl it.
The section can stay indexed in Search. The restriction applies to the uses covered by Google-Extended.
If you want to opt out of training and stay eligible to appear as a source in Gemini, Google-Extended alone doesn’t let you separate those choices.
How will the new AI control in robots.txt work?
In its 30 October 2026 announcement, Google said it plans to support a signal based on work by AIPREF, the Internet Engineering Task Force (IETF) working group on AI content usage preferences.
The technical draft, updated on 16 November, proposes the identifier __usage_ai-training. It uses existing robots.txt syntax and lets website owners express a preference for the whole site or for individual paths.
The idea is that a website can express the same preference to multiple providers, provided each one supports the signal.
There’s one detail worth checking.
If a site uses the new signal to opt out of training and doesn’t specify a rule for Google-Extended, Google plans to treat that choice as a restriction on Google-Extended.
The effect could therefore extend to grounding in the Gemini products covered by that control.
Rules specific to Google-Extended will continue to work alongside the shared signal.
For now, Google has announced support and has yet to roll it out. The signal is still described in a draft, and each provider can decide whether to adopt it.
Until then, Google’s existing controls still apply.
Does blocking Google-Extended exclude a site from AI Overviews and AI Mode?
No. Gemini Apps and the AI features in Google Search use different controls.
AI Overviews and AI Mode are part of Google Search. To appear as a supporting link, a page must be indexed and eligible to appear with a snippet.
Blocking Google-Extended doesn’t change those requirements.
If you want to limit what Google shows or reuses in its Search AI features, look at nosnippet, data-nosnippet and max-snippet.
To keep a page out of the index entirely, you can use noindex. Googlebot needs to be able to reach the page to read that directive.
Blocking the page in robots.txt as well can prevent Googlebot from seeing noindex.
How can you manage AI crawlers today?
Before changing robots.txt, separate crawl access, indexing and content use.
You might want Google to keep crawling the pages that bring in traffic and enquiries, keep those pages eligible to appear in AI answers and limit training where the provider offers a separate control.
OpenAI, for example, documents two independent settings. OAI-SearchBot is used for search in ChatGPT, while GPTBot is associated with model training.
You can allow the first and restrict the second. Google-Extended covers both training and the grounding uses described above.
For an audit, I’d start with three checks.
- Read the live file. Check the general rules, the rules for individual bots and the files on relevant subdomains. The
robots.txtfile on the main domain doesn’t automatically apply to other hosts. - Identify which sections need different rules. Public guides, product pages and original research can serve different purposes. If the URL structure allows, apply the rule to the relevant section.
- Check priority URLs after each change. The
robots.txtreport in Search Console shows the file Google fetched and any errors it found. Use URL Inspection to check whether Google can crawl your priority pages. These tools help you check crawl access for Search. They can’t tell you whether a page has been used to train a model.
Before 2027, the useful decision is which content to make available, to whom and for what purpose.
When Google starts supporting the shared signal, it’ll be easier to see where to use it and where to keep a provider-specific rule.
If you’re reviewing your site’s settings, RichClicks’ SEO consultancy can help you audit your crawl rules and identify the pages you want to keep visible in Google Search and AI search.








