Knocklog

How the 1,000 most-visited websites treat AI agents

Most of the web's biggest sites still let every AI agent in. The ones that block draw a line between training and answers: they keep their content out of model training and stay open to the assistants that cite them and send visits.

Data collected on 11 November 2026

Key findings

Who gets blocked

Sites that block each agent on the home page
AgentWhat it is forSites that block it
BytespiderTraining17.4%
CCBotTraining17%
ClaudeBotTraining16%
GPTBotTraining15.8%
Meta-ExternalAgentTraining14.4%
Applebot-ExtendedTraining14.2%
AmazonbotTraining13.5%
Google-ExtendedTraining13.4%
PerplexityBotAI search10.2%
Perplexity-UserA person's request8.7%
ChatGPT-UserA person's request8.2%
Claude-UserA person's request7.9%
Claude-SearchBotAI search7.5%
OAI-SearchBotAI search6.2%

Each company's training crawler is blocked far more often than its search crawler: GPTBot on 15.8% against 6.2% for OAI-SearchBot, ClaudeBot on 16% against 7.5% for Claude-SearchBot. Blocking a training crawler does not take a site out of assistants' answers; blocking the search and request agents does.

How we did it

We used Tranco list Y895G of the most-visited sites and took .com domains in rank order until 1,000 sites answered. 328 domains that did not answer over HTTPS within 10 seconds were left out; many are infrastructure domains with no website.

For each site we requested /robots.txt and /llms.txt once, identified as KnocklogScanner. We decided whether each agent may read the home page under RFC 9309: the group naming the agent applies, otherwise the * group. No robots.txt means everything is allowed; a server error on robots.txt counts as blocked. An llms.txt counts when it is text that starts with a title.

Limits: robots.txt states what a site asks for, not what it enforces. Blocks at the CDN or firewall do not show here. We looked at the home page only, and at .com domains only.

The data

The spreadsheet with each site's result is at knocklog.ai/research/2026-11-top-1000-robots.csv. You may use and cite it, with credit to Knocklog. We will repeat the survey every month.

See how your site treats each AI agent

The check is free and needs no sign-up.

Sources

Read next