How the 1,000 most-visited websites treat AI agents
Most of the web's biggest sites still let every AI agent in. The ones that block draw a line between training and answers: they keep their content out of model training and stay open to the assistants that cite them and send visits.
Data collected on 11 November 2026
Key findings
- 77.5% of sites let in all 14 AI agents we tested.
- 22% block at least one AI training crawler, and only 8.6% block all 8.
- GPTBot, which collects content to train OpenAI's models, is blocked on 15.8% of sites. OAI-SearchBot, which puts sites in ChatGPT search, on 6.2%.
- 9.6% block OpenAI's training and allow its search: the choice of sites that want to show up in ChatGPT without giving content to training.
- 4.1% shut the door on every AI agent, including those that fetch a page because a person asked.
- 16.9% already publish an llms.txt.
- The most blocked is Bytespider (17.4%).
Who gets blocked
| Agent | What it is for | Sites that block it |
|---|---|---|
| Bytespider | Training | |
| CCBot | Training | |
| ClaudeBot | Training | |
| GPTBot | Training | |
| Meta-ExternalAgent | Training | |
| Applebot-Extended | Training | |
| Amazonbot | Training | |
| Google-Extended | Training | |
| PerplexityBot | AI search | |
| Perplexity-User | A person's request | |
| ChatGPT-User | A person's request | |
| Claude-User | A person's request | |
| Claude-SearchBot | AI search | |
| OAI-SearchBot | AI search |
Each company's training crawler is blocked far more often than its search crawler: GPTBot on 15.8% against 6.2% for OAI-SearchBot, ClaudeBot on 16% against 7.5% for Claude-SearchBot. Blocking a training crawler does not take a site out of assistants' answers; blocking the search and request agents does.
How we did it
We used Tranco list Y895G of the most-visited sites and took .com domains in rank order until 1,000 sites answered. 328 domains that did not answer over HTTPS within 10 seconds were left out; many are infrastructure domains with no website.
For each site we requested /robots.txt and /llms.txt once, identified as KnocklogScanner. We decided whether each agent may read the home page under RFC 9309: the group naming the agent applies, otherwise the * group. No robots.txt means everything is allowed; a server error on robots.txt counts as blocked. An llms.txt counts when it is text that starts with a title.
Limits: robots.txt states what a site asks for, not what it enforces. Blocks at the CDN or firewall do not show here. We looked at the home page only, and at .com domains only.
The data
The spreadsheet with each site's result is at knocklog.ai/research/2026-11-top-1000-robots.csv. You may use and cite it, with credit to Knocklog. We will repeat the survey every month.
See how your site treats each AI agent
The check is free and needs no sign-up.
Sources
Read next
- Knocklog vs Cloudflare AI Crawl Control vs Known Agents
- How to block AI training without disappearing from ChatGPT
- What is llms.txt and how to create yours
- An RB2B alternative that also sees the AI agents
- GPTBot vs ChatGPT-User vs OAI-SearchBot: what each OpenAI bot does
- How to see AI bot traffic on WordPress
- How to tell if the GPTBot, ClaudeBot or Googlebot that visited your site is fake
- How to see AI crawler traffic on a Next.js site on Vercel or Netlify
- How to see ChatGPT traffic in Google Analytics 4
- Tools to see which AI agents and crawlers visit your website
- How to make a Lovable site visible to ChatGPT and other AI agents
- How to see which AI agents visit my website
- Directory of AI agents