robots.txt tester for AI agents

Blocking AI training without disappearing from AI answers

Most operators now use separate names for separate jobs. OpenAI has GPTBot (training), OAI-SearchBot (search results in ChatGPT) and ChatGPT-User (fetching a page for a person). Anthropic has ClaudeBot, Claude-SearchBot and Claude-User. Google uses Google-Extended for Gemini training, while Googlebot keeps your site in Search.

Blocking GPTBot or ClaudeBot stops training on your pages; blocking the search and user agents makes you vanish from the answers people read. This tester shows the effect of your rules on each one, grouped by what the agent does.

How the decision is made

Agents follow RFC 9309: an agent uses the group that names its own token; if none does, it uses the * group. Inside the group, the longest matching path wins, and Allow wins a tie. A robots.txt that answers with a server error tells agents to stay away; a missing robots.txt allows everything.

robots.txt is a request, not a lock. To see whether agents actually respect it, and to catch impostors using their names, measure real visits with Knocklog.

Questions

How do I block GPTBot in robots.txt?

Add "User-agent: GPTBot" followed by "Disallow: /". Test it here: GPTBot shows as blocked while OAI-SearchBot and ChatGPT-User stay allowed, so you still appear in ChatGPT.

Does blocking Google-Extended remove my site from Google Search?

No. Google-Extended only controls use for Gemini training and grounding. Googlebot is what keeps your pages in Search.

Why does an agent with no rule of its own show as blocked?

It falls back to the * group. If that group disallows the page, every agent without its own group is blocked too.

Do AI agents really obey robots.txt?

The major operators say their crawlers do; agents that fetch a page because a person asked often do not read it. Knocklog's report shows what each agent actually did on your site.