Knocklog

Claude-SearchBot

Anthropic's AI search crawler.

Operator
Anthropic
Type
AI search crawler
In the user agent
Claude-SearchBot
robots.txt token
Claude-SearchBot
Follows robots.txt
says it follows robots.txt
Official documentation
support.claude.com
Checked on
2026-11-04

What is Claude-SearchBot?

Claude-SearchBot is Anthropic's AI search crawler. It builds the index an AI assistant searches when it answers. If it cannot read your pages, the assistant cannot cite or recommend them.

The operator's description: Crawls the web to improve the quality of Claude's search results. (support.claude.com)

How do I block or allow Claude-SearchBot?

With robots.txt, using the token Claude-SearchBot. Anthropic says it follows robots.txt. To block the whole site:

User-agent: Claude-SearchBot
Disallow: /

To allow it, use Allow instead of Disallow. Before blocking, weigh what it brings: without it, the assistant cannot find your pages to cite or recommend them.

How do I see whether Claude-SearchBot visits my site?

In your server or CDN access logs, look for Claude-SearchBot in the user agent. Google Analytics does not show these visits: agents do not run its JavaScript.

Anyone can copy the name in a user agent, so counting log lines is not enough. Knocklog reads your site's traffic on any host, confirms who is who, and shows how often Claude-SearchBot came, which pages it read and what that brought you.

How do I know it is really Claude-SearchBot?

Anthropic publishes Claude-SearchBot's IP addresses at https://claude.com/crawling/bots.json. A visit that claims to be Claude-SearchBot from outside those ranges is spoofed.

Knocklog runs this check on every visit and separates the genuine agent from impostors.

Does Claude-SearchBot visit your site?

See in seconds how your site treats AI agents. Then connect your traffic to learn who really came.

Other Anthropic agents

All AI agents