Known bots · Common Crawl
What is CCBot?
Common Crawl’s crawler for AI training. tidekeeper knows it by its User-Agent.
- Run by
- Common Crawl
- Reads for
- AI training
- Known by
- “ccbot” in its User-Agent, and CCBot in robots.txt
- Its addresses
- Common Crawl publishes no list, so a read is known by its name alone: anyone can send it.
- Its owner’s page
- commoncrawl.org/ccbot
Keep it out
User-agent: CCBot Disallow: /
Or only part of the site: Disallow: /private/ under the same User-agent line.
On tidekeeper.eu
CCBot has not read tidekeeper.eu in the last 30 days.