Known bots · Google
Google-Extended
A name robots.txt answers to that no crawler sends. Google says where it is let in, what Googlebot reads may be used for training future Gemini models, and grounding in Gemini.
- Run by
- Reads for
- robots.txt name
- About the reads of
- Googlebot
- Its owner’s page
- developers.google.com/search/docs/crawling-indexing/google-common-crawlers
Keep it out
User-agent: Google-Extended Disallow: /
Or only part of the site: Disallow: /private/ under the same User-agent line.