Free tool · AI crawlers

Can AI read your site?

1 of 24 AI crawlers are kept out of the home pagerobots.txt has 5 groups, there is an llms.txt, and robots.txt names a sitemap.23 let in1 kept outllms.txt

Each AI crawler, by who runs it

For the home page. A crawler reads by the group that names it, else by *.

OpenAI

GPTBotAI trainingLet in

OAI-SearchBotAI searchLet in

ChatGPT-UserFor someone nowLet inOpenAI: as someone asked for it, “robots.txt rules may not apply”.

Anthropic

ClaudeBotAI trainingLet in

Claude-SearchBotAI searchLet in

Claude-UserFor someone nowLet in

Perplexity

PerplexityBotAI searchLet in

Perplexity-UserFor someone nowLet in

Mistral AI

MistralAI-UserFor someone nowLet in

MistralAI-IndexAI searchLet in

MistralAI-TrainingAI trainingLet in

Amazon

AmazonbotAI trainingLet in

ByteDance

BytespiderAI trainingKept outbytespider · Disallow: /

Common Crawl

CCBotAI trainingLet in

Cohere

cohere-aiAI trainingLet in

DuckDuckGo

DuckAssistBotAI searchLet in

You.com

YouBotAI searchLet in

Google

GooglebotSearch engineLet in

Google-Extendedrobots.txt nameLet inCrawls nothing itself: says whether what Googlebot reads may be used for training future Gemini models, and grounding in Gemini.

Apple

ApplebotSearch engineLet in

Applebot-Extendedrobots.txt nameLet inCrawls nothing itself: says whether what Applebot reads may be used for training Apple's foundation models for its generative AI features.

What robots.txt says

github.com/robots.txt

# If you would like to crawl GitHub contact us via https://support.github.com?tags=dotcom-robots
# We also provide an extensive API: https://docs.github.com
Sitemap: https://github.com/sitemap_marketing.xml

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: PerplexityBot
Crawl-delay: 1
Allow: /*?tab=achievements&achievement=*
Allow: /marketplace/*?tab=readme$
Allow: /marketplace/*?tab=transparency$
Disallow: /account-login
Disallow: /copilot/
Disallow: /copilot/c/
Disallow: /ekansa/Open-Context-Data
Disallow: /ekansa/opencontext-*
Disallow: /Explodingstuff/
Disallow: /gist/
Disallow: /search/advanced
Disallow: /search$
Disallow: /*/*/pulse_committer_data.json$
Disallow: /*/*/pulse_committer_data.json?*
Disallow: /*/archive/
Disallow: /*/blame/
Disallow: /*/cache/
Disallow: /*/raw/
Disallow: /*/tree/
Disallow: */tarball/
Disallow: */zipball/
Disallow: /.git/
Disallow: */.git/
Disallow: /*.git$
Disallow: /*.atom$
Disallow: /*report-abuse?report=*
Disallow: /*/*/commits/
Disallow: /marketplace/*/*?tab=readme$
Disallow: /marketplace/*/*?tab=transparency$
Disallow: /marketplace/*?*?tab=readme$
Disallow: /marketplace/*?*?tab=transparency$
Disallow: /marketplace/actions?tab=readme$
Disallow: /marketplace/actions?tab=transparency$
Disallow: /marketplace/free-trials?tab=readme$
Disallow: /marketplace/free-trials?tab=transparency$
Disallow: /marketplace/manage?tab=readme$
Disallow: /marketplace/manage?tab=transparency$
Disallow: /marketplace/models?tab=readme$
Disallow: /marketplace/models?tab=transparency$
Disallow: /marketplace/new?tab=readme$
Disallow: /marketplace/new?tab=transparency$
Disallow: /marketplace/publishers?tab=readme$
Disallow: /marketplace/publishers?tab=transparency$
Disallow: /marketplace/recently-added?tab=readme$
Disallow: /marketplace/recently-added?tab=transparency$
Disallow: /biztools$
Disallow: /biztools?*
Disallow: /biztools/*
Disallow: /businesses$
Disallow: /businesses?*
Disallow: /businesses/*
Disallow: /enterprises$
Disallow: /enterprises?*
Disallow: /enterprises/*
Disallow: /organizations$
Disallow: /organizations?*
Disallow: /organizations/*
Disallow: /settings$
Disallow: /settings?*
Disallow: /settings/*
Disallow: /stafftools$
Disallow: /stafftools?*
Disallow: /stafftools/*
Disallow: /users/*/billing$
Disallow: /users/*/billing?*
Disallow: /users/*/billing/*
Disallow: /writing$
Disallow: /writing?*
Disallow: /writing/*
Disallow: /*/*/branches$
Disallow: /*/*/branches?*
Disallow: /*/*/branches/*
Disallow: /*/*/comments$
Disallow: /*/*/comments?*
Disallow: /*/*/comments/*
Disallow: /*/*/compare$
Disallow: /*/*/compare?*
Disallow: /*/*/compare/*
Disallow: /*/*/contributors$
Disallow: /*/*/contributors?*
Disallow: /*/*/contributors/*
Disallow: /*/*/contributors_list$
Disallow: /*/*/contributors_list?*
Disallow: /*/*/contributors_list/*
Disallow: /*/*/contributors-list$
Disallow: /*/*/contributors-list?*
Disallow: /*/*/contributors-list/*
Disallow: /*/*/download$
Disallow: /*/*/download?*
Disallow: /*/*/download/*
Disallow: /*/*/forks$
Disallow: /*/*/forks?*
Disallow: /*/*/forks/*
Disallow: /*/*/graphs$
Disallow: /*/*/graphs?*
Disallow: /*/*/graphs/*
Disallow: /*/*/issues/new$
Disallow: /*/*/issues/new?*
Disallow: /*/*/issues/new/*
Disallow: /*/*/issues/search$
Disallow: /*/*/issues/search?*
Disallow: /*/*/issues/search/*
Disallow: /*/*/milestones/new$
Disallow: /*/*/milestones/new?*
Disallow: /*/*/milestones/new/*
Disallow: /*/*/network$
Disallow: /*/*/network?*
Disallow: /*/*/network/*
Disallow: /*/*/projects$
Disallow: /*/*/projects?*
Disallow: /*/*/projects/*
Disallow: /*/*/pulse$
Disallow: /*/*/pulse?*
Disallow: /*/*/pulse/*
Disallow: /*/*/pulse-new$
Disallow: /*/*/pulse-new?*
Disallow: /*/*/pulse-new/*
Disallow: /*/*/pulse_committer_data$
Disallow: /*/*/pulse_committer_data?*
Disallow: /*/*/pulse_committer_data/*
Disallow: /*/*/pulse_diffstat_summary$
Disallow: /*/*/pulse_diffstat_summary?*
Disallow: /*/*/pulse_diffstat_summary/*
Disallow: /*/*/revi

Not every crawler keeps robots.txt. On your own site, tidekeeper sees who read what anyway.

To keep the AI trainers out

Add to robots.txt

# Training only: search and assistants still read
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: MistralAI-Training
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: Bytespider
User-agent: CCBot
User-agent: cohere-ai
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

Lines for the crawlers that read for training, as tidekeeper knows them. Search engines and the assistants that read for someone, now, are left as they are.

The page, as a crawler that runs no script gets it

200525.4 KB of HTML1137 words146 links

Title: GitHub · Change is constant. GitHub keeps you ahead. · GitHub · GitHub
First heading: Software still starts with you
Description: Join the world's most widely adopted, AI-powered developer platform where millions of developers, businesses, and the largest open source community build software that advances humanity.
The first words: GitHub Universe ‘26 Recap Software still starts with you Enter your email Sign up for GitHub Download GitHub Copilot app Product walkthrough Copilot reviews proposed code changes. GitHub activity push from Pankaj2004s Pankaj2004s/fs-01-shopsphere · Nov 5, 2026 225M+ 225M+ developers

Then, which of them actually came

robots.txt says who may read. tidekeeper's Bots and AI says who did: each crawler by name, the pages it read, whether its address was its owner's, and the readers each assistant sent after.

Bots and AI crawlers →

How it works

Give a domainIts robots.txt, its llms.txt and the page are fetched once.
See each crawlerBy company: let in or kept out, and by which rule.
Keep out what you chooseCopy the lines that keep the crawlers that train out, and leave search and assistants be.

Questions

How do I block GPTBot?

Put “User-agent: GPTBot” and “Disallow: /” in your robots.txt. OpenAI's search reads as OAI-SearchBot, by its own name, so ChatGPT can still find your pages.

What is Google-Extended?

A name in robots.txt, not a crawler: Googlebot does the reading, and Google-Extended says whether what it reads may be used for Gemini and Vertex AI.

Does blocking AI crawlers change my Google ranking?

Keeping Google-Extended out does not change Google Search, Google says: Googlebot reads for search under its own name.

What is llms.txt?

A Markdown file at /llms.txt that says what a site is and where its pages are as text, for AI agents. It is a proposal, read by some agents.

Do crawlers keep robots.txt?

The ones that say so do; not every crawler does. On your own site, tidekeeper sees which ones came, and what they read.