OpenAI read our site 524 times in four days. ChatGPT sent us no one.
From 23 to 26 October, bots made 1,947 requests to tidekeeper.eu, eight for every page a person viewed. Who they were, what they read, what came back, and the three things they found wrong on our side.
- In four days, bots made 1,947 requests to our site. People viewed 242 pages.
- Almost half came from crawlers gathering text to train AI; OpenAI's GPTBot alone made 511. No AI assistant sent us a single visitor.
- The bots found two broken addresses in the list we keep for AI agents, and a link that misled one of them. All three are fixed.
Since 23 October, the server behind tidekeeper.eu has told tidekeeper about every request from a bot it knows by name. Most crawlers read a page and leave without running its script, so the script alone never sees them.1 Here are the first four whole days, 23 to 26 October, beside the people our script counted in the same days.
Who they were
Seventeen bots gave their names. Five of them we could check against the addresses their owners publish for their crawlers; one name was sometimes a disguise, as below; the rest we take at their word.
Almost half of all requests, 875, came from crawlers gathering text to train AI models: GPTBot, Amazonbot, Anthropic's ClaudeBot and ByteDance's Bytespider. OpenAI describes GPTBot as crawling content "that may be used in training" its models.2 A third, 646, came from SEO tools, which map the web's links and sell the map to marketers. Search engines made 367.
| What the request was for | Requests | Share |
|---|---|---|
| Gathering text to train AI | 875 | 45% |
| Mapping links for SEO tools | 646 | 33% |
| A search engine's index | 367 | 19% |
| A preview of a shared link | 42 | 2% |
| Finding pages for an assistant's answers | 17 | 1% |
| Reading a page because a person just asked an assistant about it | 0 | 0% |
What came back
The last row is the one we watch. A training crawler reads to learn. An assistant's search bot reads so that an answer can cite the page. The third kind fetches a page because someone asked ChatGPT or Perplexity about it a moment ago: a reader at one remove. In four days there were none, and nobody arrived from an assistant either.
In the same days Googlebot asked for six addresses, and Google's search sent us 12 visitors. A search engine reads to send people. A training crawler reads to learn, and whoever learns from it leaves no trace we can see: an answer that names tidekeeper without a link shows up in nobody's analytics, ours included.
Most of what the bots read was the same few files. Our robots.txt, where a site says what crawlers may read, was asked for 111 times, and the sitemap 44 times. Ours lets every crawler read every page; only the API, and the address our script reports to, are closed to them. Then came pricing (71) and the front page (70), and after them our API reference, a page for each endpoint, as a page and as Markdown.
What they found for us
Watching bots read a site is also a test of the site. Ours failed in three places.
- Two of our own addresses were broken.
/llms.txtis the list of Markdown pages a site keeps for AI agents.3 Ours listed/docs/bots.mdand/docs/stores.md, and both answered 404: bots asked for them nine times and got nothing. Our server added each Markdown page by hand, and those two pages came later. Now the addresses come from the same list/llms.txtis made from, and a test asks for every one it lists. - One of our links misled a crawler. Each page of our docs has links that ask ChatGPT, Claude or Perplexity about it, with the question inside the link, the page's address readable in the middle. An AI crawler took the address from there with the rest of the sentence attached, asked for
/docs/api/get-about.md%20so%20I%20can%20ask%20questions%20about%20it., and got a 404, for page after page of our API reference. The address inside the question is now encoded whole.
And some of them weren't who they said. Twenty-seven requests called themselves Googlebot; two came from addresses that aren't on Google's lists. Google publishes how to tell its crawlers from the rest.4 A bot's name is a claim, and its address is the check.
What we take from it
So far AI reads us and sends no one back, and that is worth knowing rather than guessing. We won't block the crawlers: if an assistant ever answers with a link to us, it will be because one of them read us first. What we changed is the part we control. The pages they ask for now answer, and our links say what they mean. The day an assistant does send someone, they will show up beside its crawler's reads.
- tidekeeper.eu's own figures, Bots and AI and the dashboard, 23 to 26 October 2026, days in UTC; how the server tells tidekeeper what bots read is in Bots and AI crawlers.
- OpenAI, Overview of OpenAI Crawlers: GPTBot, OAI-SearchBot and ChatGPT-User, what each is for, and the addresses each uses.
- Jeremy Howard, The /llms.txt file (2024), the proposal.
- Google Search Central, Verify requests from Google crawlers and fetchers.
# OpenAI read our site 524 times in four days. ChatGPT sent us no one. > tidekeeper's blog, 28 October 2026, by Christina Sørensen: https://tidekeeper.eu/blog/who-reads-our-site From 23 to 26 October, bots made 1,947 requests to tidekeeper.eu, eight for every page a person viewed. Who they were, what they read, what came back, and the three things they found wrong on our side. ## In short - In four days, bots made 1,947 requests to our site. People viewed 242 pages. - Almost half came from crawlers gathering text to train AI; OpenAI's GPTBot alone made 511. No AI assistant sent us a single visitor. - The bots found two broken addresses in the list we keep for AI agents, and a link that misled one of them. All three are fixed. Since 23 October, the server behind tidekeeper.eu has told tidekeeper about every request from a bot it knows by name. Most crawlers read a page and leave without running its script, so the script alone never sees them.[1] Here are the first four whole days, 23 to 26 October, beside the people our script counted in the same days. | | | |---|---| | Requests by bots | 1,947 (17 bots, by name) | | Pages people viewed | 242 (122 visitors, summed by day) | | Bot requests for each | 8 (page a person viewed) | | Visitors sent by an AI | 0 (ChatGPT, Perplexity, Claude or any other) | Figure 1. tidekeeper.eu, 23 to 26 October 2026, days in UTC. Source: tidekeeper.eu's own figures: Bots and AI for the bots, the dashboard for the people. ## Who they were Seventeen bots gave their names. Five of them we could check against the addresses their owners publish for their crawlers; one name was sometimes a disguise, as below; the rest we take at their word. | | | | |---|---|---| | GPTBot, OpenAI | 511 | AI training | | MJ12bot, Majestic | 499 | SEO tool | | Applebot, Apple | 335 | search engine | | Amazonbot, Amazon | 288 | AI training | | SemrushBot, Semrush | 120 | SEO tool | | ClaudeBot, Anthropic | 74 | AI training | | Twitterbot, X | 28 | link previews | | Googlebot, Google | 27 | search engine | | nine more | 65 | 2 to 27 each | 511 requests from GPTBot alone, OpenAI's crawler for training data. Figure 2. Requests by bot, 23 to 26 October. Source: tidekeeper.eu's own figures, Bots and AI. Almost half of all requests, 875, came from crawlers gathering text to train AI models: GPTBot, Amazonbot, Anthropic's ClaudeBot and ByteDance's Bytespider. OpenAI describes GPTBot as crawling content "that may be used in training" its models.[2] A third, 646, came from SEO tools, which map the web's links and sell the map to marketers. Search engines made 367. | What the request was for | Requests | Share | |---|---|---| | Gathering text to train AI | 875 | 45% | | Mapping links for SEO tools | 646 | 33% | | A search engine's index | 367 | 19% | | A preview of a shared link | 42 | 2% | | Finding pages for an assistant's answers | 17 | 1% | | Reading a page because a person just asked an assistant about it | 0 | 0% | Figure 3. What the requests were for. Source: tidekeeper.eu's own figures, Bots and AI; each bot's purpose as its owner describes it. ## What came back The last row is the one we watch. A training crawler reads to learn. An assistant's search bot reads so that an answer can cite the page. The third kind fetches a page because someone asked ChatGPT or Perplexity about it a moment ago: a reader at one remove. In four days there were none, and nobody arrived from an assistant either. 1. 524: requests from OpenAI's crawlers 2. 13: of them from OAI-SearchBot, which finds pages for ChatGPT's answers 3. 0: fetches by ChatGPT-User, for a person asking about a page 4. 0: visitors who came from ChatGPT Figure 4. OpenAI's crawlers, and what came back. Source: tidekeeper.eu's own figures, Bots and AI and Sources, 23 to 26 October; what each crawler is for, from OpenAI. In the same days Googlebot asked for six addresses, and Google's search sent us 12 visitors. A search engine reads to send people. A training crawler reads to learn, and whoever learns from it leaves no trace we can see: an answer that names tidekeeper without a link shows up in nobody's analytics, ours included. Most of what the bots read was the same few files. Our `robots.txt`, where a site says what crawlers may read, was asked for 111 times, and the sitemap 44 times. Ours lets every crawler read every page; only the API, and the address our script reports to, are closed to them. Then came pricing (71) and the front page (70), and after them our API reference, a page for each endpoint, as a page and as Markdown. ## What they found for us Watching bots read a site is also a test of the site. Ours failed in three places. 1. **Two of our own addresses were broken.** `/llms.txt` is the list of Markdown pages a site keeps for AI agents.[3] Ours listed `/docs/bots.md` and `/docs/stores.md`, and both answered 404: bots asked for them nine times and got nothing. Our server added each Markdown page by hand, and those two pages came later. Now the addresses come from the same list `/llms.txt` is made from, and a test asks for every one it lists. 2. **One of our links misled a crawler.** Each page of our docs has links that ask ChatGPT, Claude or Perplexity about it, with the question inside the link, the page's address readable in the middle. An AI crawler took the address from there with the rest of the sentence attached, asked for `/docs/api/get-about.md%20so%20I%20can%20ask%20questions%20about%20it.`, and got a 404, for page after page of our API reference. The address inside the question is now encoded whole. And some of them weren't who they said. Twenty-seven requests called themselves Googlebot; two came from addresses that aren't on Google's lists. Google publishes how to tell its crawlers from the rest.[4] A bot's name is a claim, and its address is the check. ## What we take from it So far AI reads us and sends no one back, and that is worth knowing rather than guessing. We won't block the crawlers: if an assistant ever answers with a link to us, it will be because one of them read us first. What we changed is the part we control. The pages they ask for now answer, and our links say what they mean. The day an assistant does send someone, they will show up beside its crawler's reads. The film, ChatGPT is reading your site. Is it sending anyone back? (5:17): https://youtu.be/BNcSkoKIV1Q ## Sources 1. tidekeeper.eu's own figures, Bots and AI and the dashboard, 23 to 26 October 2026, days in UTC; how the server tells tidekeeper what bots read is in [Bots and AI crawlers](https://tidekeeper.eu/docs/bots). 2. OpenAI, [Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots): GPTBot, OAI-SearchBot and ChatGPT-User, what each is for, and the addresses each uses. 3. Jeremy Howard, [The /llms.txt file](https://llmstxt.org) (2024), the proposal. 4. Google Search Central, [Verify requests from Google crawlers and fetchers](https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot).