Can AI read your site?
See, for GPTBot, ClaudeBot, PerplexityBot and every other AI crawler, what your robots.txt lets it read, whether there is an llms.txt, and the page as a crawler that runs no script gets it.
- Free, no account
- Sets no cookie
- Keeps nothing you type
Then, which of them actually came
robots.txt says who may read. tidekeeper's Bots and AI says who did: each crawler by name, the pages it read, whether its address was its owner's, and the readers each assistant sent after.
Bots and AI crawlers →How it works
- Give a domainIts robots.txt, its llms.txt and the page are fetched once.
- See each crawlerBy company: let in or kept out, and by which rule.
- Keep out what you chooseCopy the lines that keep the crawlers that train out, and leave search and assistants be.
Questions
How do I block GPTBot?
Put “User-agent: GPTBot” and “Disallow: /” in your robots.txt. OpenAI's search reads as OAI-SearchBot, by its own name, so ChatGPT can still find your pages.
What is Google-Extended?
A name in robots.txt, not a crawler: Googlebot does the reading, and Google-Extended says whether what it reads may be used for Gemini and Vertex AI.
Does blocking AI crawlers change my Google ranking?
Keeping Google-Extended out does not change Google Search, Google says: Googlebot reads for search under its own name.
What is llms.txt?
A Markdown file at /llms.txt that says what a site is and where its pages are as text, for AI agents. It is a proposal, read by some agents.
Do crawlers keep robots.txt?
The ones that say so do; not every crawler does. On your own site, tidekeeper sees which ones came, and what they read.