Bytespider

ByteDance's AI training crawler.

Operator

ByteDance

Type

AI training crawler

In the user agent

Bytespider

robots.txt token

Bytespider

Follows robots.txt

does not say whether it follows robots.txt

Official documentation

not found; identified by its observed user agent

Checked on

2026-11-04

What is Bytespider?

Bytespider is ByteDance's AI training crawler. It collects pages that may be used to train AI models. A visit does not answer anyone right now; it decides what future models know about you.

The operator's description: ByteDance crawler associated with AI model data collection.

How do I block or allow Bytespider?

With robots.txt, using the token Bytespider. ByteDance does not say whether it follows robots.txt. To block the whole site:

User-agent: Bytespider
Disallow: /

To allow it, use Allow instead of Disallow. Before blocking, weigh what it brings: blocking does not remove you from AI search, but future models will know less about you.

How do I see whether Bytespider visits my site?

In your server or CDN access logs, look for Bytespider in the user agent. Google Analytics does not show these visits: agents do not run its JavaScript.

Anyone can copy the name in a user agent, so counting log lines is not enough. Knocklog reads your site's traffic on any host, confirms who is who, and shows how often Bytespider came, which pages it read and what that brought you.

How do I know it is really Bytespider?

ByteDance does not publish a way to verify Bytespider. Any program can use that name, so Knocklog shows these visits as “unverified”, never as genuine.

All AI agents