Bots and AI crawlers
This page as Markdown, or in an AI chatMore ways to read this page
Who reads your site that is not a person: each bot by its name, what it reads for, the pages it reads, and whether it is who it says. Never counted as a view or a visitor.
# Bots and AI crawlers
> tidekeeper's documentation: https://tidekeeper.eu/docs/bots
Who reads your site that is not a person: each bot by its name, what it reads for, the pages it reads, and whether it is who it says. Never counted as a view or a visitor.
## Why the script is not enough
A search engine's crawler and most AI crawlers, GPTBot, ClaudeBot, PerplexityBot and the rest, read a page and leave: they never run its script. The few that do are kept by their names from the script, and left out of every figure. To see the rest, the site's own server tells tidekeeper what they read.
## From nginx, as it happens
Each site's Install screen has the lines with its name and its `READS_KEY` filled in, and makes the key: it may send what bots read for that site and nothing else. A map names the known bots by their User-Agents; a mirror sends each of their requests to tidekeeper. Anyone else's request stops in your server and is never sent.
```nginx
# in the http block, once
map $http_user_agent $tidekeeper_bot {
default 0;
"~*(gptbot|oai-searchbot|chatgpt-user|claudebot|claude-user|perplexitybot|googlebot|bingbot)" 1;
}
# in the site's server block
mirror /tidekeeper-reads;
location = /tidekeeper-reads {
internal;
if ($tidekeeper_bot = 0) { return 204; }
proxy_pass https://tidekeeper.eu/api/v1/sites/example.org/reads;
proxy_pass_request_body off;
proxy_set_header Content-Length "";
proxy_set_header Cookie "";
proxy_set_header Authorization "Bearer READS_KEY";
proxy_set_header X-Tidekeeper-Path $request_uri;
proxy_set_header X-Tidekeeper-Address $remote_addr;
access_log off;
}
```
The Install screen's map names every bot tidekeeper knows; this one is shortened. A request from a bot it does not know is dropped unread.
## Or from the log nginx already writes
On the server, with the reads key or a token that may send reads, `--bots` sends each line a known bot wrote, with its answer: what was there, and what was asked for and was not, `robots.txt` and `llms.txt` among them. The mirror cannot see an answer; the log can.
```sh
tidekeeper import nginx /var/log/nginx/access.log --site example.org --follow --bots
```
## Who it is
OpenAI, Google, Microsoft, Apple, DuckDuckGo and Perplexity publish the addresses their bots read from. tidekeeper fetches those lists once a day and checks each read against its bot's: checked, or not who it says, when someone else uses the bot's name. A bot whose owner publishes no list is known by its name only, and says so.
## What is kept
- The bot's name, the page, the day, how many reads and the last, the answer where the server said, and whether it was checked.
- Not the address: it is used once to check the bot against its owner's list, then dropped.
- Kept as long as the site's other counts, and gone with the site.
## Counted in
A switch on the Bots and AI screen, off at first. On, the overview draws what bots read as a series of its own under its chart, Pages has a column for it, and every screen says bots are counted in. Views and visitors never include a bot.
## From the API and an agent
`GET /api/v1/sites/{site}/bots` answers what the screen shows; the MCP server's `bots` tool asks it.
Why the script is not enough
A search engine's crawler and most AI crawlers, GPTBot, ClaudeBot, PerplexityBot and the rest, read a page and leave: they never run its script. The few that do are kept by their names from the script, and left out of every figure. To see the rest, the site's own server tells tidekeeper what they read.
From nginx, as it happens
Each site's Install screen has the lines with its name and its READS_KEY filled in, and makes the key: it may send what bots read for that site and nothing else. A map names the known bots by their User-Agents; a mirror sends each of their requests to tidekeeper. Anyone else's request stops in your server and is never sent.
# in the http block, once map $http_user_agent $tidekeeper_bot { default 0; "~*(gptbot|oai-searchbot|chatgpt-user|claudebot|claude-user|perplexitybot|googlebot|bingbot)" 1; } # in the site's server block mirror /tidekeeper-reads; location = /tidekeeper-reads { internal; if ($tidekeeper_bot = 0) { return 204; } proxy_pass https://tidekeeper.eu/api/v1/sites/example.org/reads; proxy_pass_request_body off; proxy_set_header Content-Length ""; proxy_set_header Cookie ""; proxy_set_header Authorization "Bearer READS_KEY"; proxy_set_header X-Tidekeeper-Path $request_uri; proxy_set_header X-Tidekeeper-Address $remote_addr; access_log off; }
The Install screen's map names every bot tidekeeper knows; this one is shortened. A request from a bot it does not know is dropped unread.
Or from the log nginx already writes
On the server, with the reads key or a token that may send reads, --bots sends each line a known bot wrote, with its answer: what was there, and what was asked for and was not, robots.txt and llms.txt among them. The mirror cannot see an answer; the log can.
tidekeeper import nginx /var/log/nginx/access.log --site example.org --follow --bots
Who it is
OpenAI, Google, Microsoft, Apple, DuckDuckGo and Perplexity publish the addresses their bots read from. tidekeeper fetches those lists once a day and checks each read against its bot's: checked, or not who it says, when someone else uses the bot's name. A bot whose owner publishes no list is known by its name only, and says so.
What is kept
- The bot's name, the page, the day, how many reads and the last, the answer where the server said, and whether it was checked.
- Not the address: it is used once to check the bot against its owner's list, then dropped.
- Kept as long as the site's other counts, and gone with the site.
Counted in
A switch on the Bots and AI screen, off at first. On, the overview draws what bots read as a series of its own under its chart, Pages has a column for it, and every screen says bots are counted in. Views and visitors never include a bot.
From the API and an agent
GET /api/v1/sites/{site}/bots answers what the screen shows; the MCP server's bots tool asks it.