Known bots · Google
What is Googlebot?
Google’s search engine's crawler. tidekeeper knows it by its User-Agent, and checks each read against the addresses Google publishes.
- Run by
- Reads for
- Search engine
- Known by
- “googlebot” in its User-Agent, and Googlebot in robots.txt
- Its addresses
- Google publishes them: developers.google.com/static/crawling/ipranges/common-crawlers.json. tidekeeper checks every read against the list.
- Its owner’s page
- developers.google.com/search/docs/crawling-indexing/googlebot
Keep it out
User-agent: Googlebot Disallow: /
Or only part of the site: Disallow: /private/ under the same User-agent line.
On tidekeeper.eu
Googlebot read tidekeeper.eu 55 times in the last 30 days, as tidekeeper counts its own site, from its server.