Most of the web's biggest sites still let every AI agent in
AI companies send crawlers to train models, crawlers to answer search queries, and agents that browse for a user. Site owners can treat each one differently in robots.txt. We checked how the 1,000 most-visited .com sites do it: three in four leave the door open to all of them, and the few that block mostly target training, not AI search.
- 77.5% of the 1,000 most-visited .com sites allow every one of the 14 major AI agents we tested on their home page.
- 22% block at least one AI training crawler; 8.6% block all eight we tested.
- The most-blocked agents are ByteDance's Bytespider (17.4%), Common Crawl's CCBot (17%), Anthropic's ClaudeBot (16%) and OpenAI's GPTBot (15.8%).
- 9.6% block GPTBot, which collects training data, while still allowing OAI-SearchBot, which powers ChatGPT search.
- Only 16.9% publish an llms.txt file, the proposed guide for AI assistants.
- Method: Tranco top-sites list Y895G, .com domains in rank order, robots.txt read with the RFC 9309 rules on 11 November 2026.