Known bots · Common Crawl
Common Crawl’s bots
Common Crawl runs one bot tidekeeper knows by name.
Its bots
Common Crawl’s own page on them: commoncrawl.org/ccbot
Keep one out
A crawler reads by the group that names it. To keep it out:
User-agent: CCBot Disallow: /