CCBot

Common Crawl's AI training crawler.

Operator

Common Crawl

Type

AI training crawler

In the user agent

CCBot

robots.txt token

CCBot

Follows robots.txt

does not say whether it follows robots.txt

Official documentation

not found; identified by its observed user agent

Checked on

2026-11-04

What is CCBot?

CCBot is Common Crawl's AI training crawler. It collects pages that may be used to train AI models. A visit does not answer anyone right now; it decides what future models know about you.

The operator's description: Builds the Common Crawl open web archive, widely used as AI training data.

How do I block or allow CCBot?

With robots.txt, using the token CCBot. Common Crawl does not say whether it follows robots.txt. To block the whole site:

User-agent: CCBot
Disallow: /

To allow it, use Allow instead of Disallow. Before blocking, weigh what it brings: blocking does not remove you from AI search, but future models will know less about you.

How do I see whether CCBot visits my site?

In your server or CDN access logs, look for CCBot in the user agent. Google Analytics does not show these visits: agents do not run its JavaScript.

Anyone can copy the name in a user agent, so counting log lines is not enough. Knocklog reads your site's traffic on any host, confirms who is who, and shows how often CCBot came, which pages it read and what that brought you.

How do I know it is really CCBot?

Common Crawl does not publish a way to verify CCBot. Any program can use that name, so Knocklog shows these visits as “unverified”, never as genuine.

All AI agents