Known bots · Google

Google’s bots

Google runs 2 bots tidekeeper knows by name, each with its own name in robots.txt.

Its bots

GooglebotSearch engineaddress listread us 55× this monthGoogle-Extendedrobots.txt namea robots.txt namereads nothing itself

Google’s own pages on them: developers.google.com/search/docs/crawling-indexing/googlebot, developers.google.com/search/docs/crawling-indexing/google-common-crawlers

Keep one out

Each reads by the group that names it. To keep Googlebot out and let the others read:

robots.txt
User-agent: Googlebot
Disallow: /

User-agent: Google-Extended
Allow: /
See which of them read your site: Bots and AI crawlers →