Known bots · Semrush
Semrush’s bots
Semrush runs one bot tidekeeper knows by name.
Its bots
Semrush’s own page on them: www.semrush.com/bot/
Keep one out
A crawler reads by the group that names it. To keep it out:
User-agent: SemrushBot Disallow: /
Semrush runs one bot tidekeeper knows by name.
Semrush’s own page on them: www.semrush.com/bot/
A crawler reads by the group that names it. To keep it out:
User-agent: SemrushBot Disallow: /