Known bots · Meta
What is Meta-WebIndexer?
Meta’s crawler for its AI search. tidekeeper knows it by its User-Agent.
- Run by
- Meta
- Reads for
- AI search
- Known by
- “meta-webindexer” in its User-Agent, and Meta-WebIndexer in robots.txt
- Its addresses
- Meta publishes no list, so a read is known by its name alone: anyone can send it.
- Its owner’s page
- developers.facebook.com/docs/sharing/webmasters/web-crawlers
Keep it out
User-agent: Meta-WebIndexer Disallow: /
Or only part of the site: Disallow: /private/ under the same User-agent line.
On tidekeeper.eu
Meta-WebIndexer has not read tidekeeper.eu in the last 30 days.