PricingGet 14 days free
Free tool · AI crawlers

Can AI read your site?

Trytidekeeper.eunytimes.comgithub.com
Every AI crawler may read the home pagerobots.txt has 1 group, there is an llms.txt, and robots.txt names a sitemap.24 let in0 kept outllms.txt

Each AI crawler, by who runs it

For the home page. A crawler reads by the group that names it, else by *.

OpenAI

GPTBotAI trainingLet in
OAI-SearchBotAI searchLet in
ChatGPT-UserFor someone nowLet inOpenAI: as someone asked for it, “robots.txt rules may not apply”.

Anthropic

ClaudeBotAI trainingLet in
Claude-SearchBotAI searchLet in
Claude-UserFor someone nowLet in

Perplexity

PerplexityBotAI searchLet in
Perplexity-UserFor someone nowLet in

Mistral AI

MistralAI-UserFor someone nowLet in
MistralAI-IndexAI searchLet in
MistralAI-TrainingAI trainingLet in

Meta

Meta-ExternalAgentAI trainingLet in
Meta-ExternalFetcherFor someone nowLet in
Meta-WebIndexerAI searchLet in

Amazon

AmazonbotAI trainingLet in

ByteDance

BytespiderAI trainingLet in

Common Crawl

CCBotAI trainingLet in

Cohere

cohere-aiAI trainingLet in

DuckDuckGo

DuckAssistBotAI searchLet in

You.com

YouBotAI searchLet in

Google

GooglebotSearch engineLet in
Google-Extendedrobots.txt nameLet inCrawls nothing itself: says whether what Googlebot reads may be used for training future Gemini models, and grounding in Gemini.

Apple

ApplebotSearch engineLet in
Applebot-Extendedrobots.txt nameLet inCrawls nothing itself: says whether what Applebot reads may be used for training Apple's foundation models for its generative AI features.

What robots.txt says

tidekeeper.eu/robots.txt
User-agent: *
Disallow: /api/
Disallow: /t/

Sitemap: https://tidekeeper.eu/sitemap.xml

Not every crawler keeps robots.txt. On your own site, tidekeeper sees who read what anyway.

To keep the AI trainers out

Add to robots.txt
# Training only: search and assistants still read
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: MistralAI-Training
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: Bytespider
User-agent: CCBot
User-agent: cohere-ai
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

Lines for the crawlers that read for training, as tidekeeper knows them. Search engines and the assistants that read for someone, now, are left as they are.

The page, as a crawler that runs no script gets it

20057.9 KB of HTML1787 words51 links
Title
tidekeeper
First heading
Analytics that grows up with you.
Description
Web analytics that counts every visitor from the first day, with no cookies and nothing kept on their device: funnels inside a visit, revenue by source, events. Retention and paths across visits when you need them. Hosted in the EU.
The first words
Web analytics · hosted in the EU Analytics that grows up with you. Count every visitor from the first day, with no cookies and nothing kept on their device: funnels inside a visit, revenue by source, events and their properties.

Then, which of them actually came

robots.txt says who may read. tidekeeper's Bots and AI says who did: each crawler by name, the pages it read, whether its address was its owner's, and the readers each assistant sent after.

Bots and AI crawlers →

How it works

  1. Give a domainIts robots.txt, its llms.txt and the page are fetched once.
  2. See each crawlerBy company: let in or kept out, and by which rule.
  3. Keep out what you chooseCopy the lines that keep the crawlers that train out, and leave search and assistants be.

Questions

How do I block GPTBot?

Put “User-agent: GPTBot” and “Disallow: /” in your robots.txt. OpenAI's search reads as OAI-SearchBot, by its own name, so ChatGPT can still find your pages.

What is Google-Extended?

A name in robots.txt, not a crawler: Googlebot does the reading, and Google-Extended says whether what it reads may be used for Gemini and Vertex AI.

Does blocking AI crawlers change my Google ranking?

Keeping Google-Extended out does not change Google Search, Google says: Googlebot reads for search under its own name.

What is llms.txt?

A Markdown file at /llms.txt that says what a site is and where its pages are as text, for AI agents. It is a proposal, read by some agents.

Do crawlers keep robots.txt?

The ones that say so do; not every crawler does. On your own site, tidekeeper sees which ones came, and what they read.