How to block AI training without disappearing from ChatGPT

Blocking every AI agent takes your site out of ChatGPT, Claude and Perplexity answers. You can split it: refuse model training and keep being found and cited.

Training and search are different agents

AI companies use a different name for each job. Training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot, Bytespider) collect pages for future models. Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) build the index the assistant searches. Assistants acting for a person (ChatGPT-User, Claude-User, Perplexity-User) fetch a page the moment someone asks.

Blocking the first group and allowing the other two is the most common policy for businesses: your content does not become training data, but your site keeps showing up and being cited with a link.

The robots.txt

Copy it into the robots.txt file at the root of your site and change the sitemap address. The names come from our agent directory, checked against each company's documentation.

# Block AI model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
User-agent: Bytespider
Disallow: /

# Allow AI search and assistants
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: PerplexityBot
User-agent: meta-externalfetcher
User-agent: meta-webindexer
User-agent: Amazonbot
User-agent: PetalBot
User-agent: DuckAssistBot
User-agent: MistralAI-User
Allow: /

# Everyone else, as today
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

What changes and what does not

Google-Extended does not remove your site from Google Search: according to Google, it controls whether content is used to train and ground Gemini. Googlebot keeps indexing as usual.
Applebot-Extended refuses training of Apple's models; Applebot keeps feeding Siri and Spotlight search.
Assistants fetching a page for a person do not always follow robots.txt: Perplexity-User generally ignores it and ChatGPT-User follows it only in part. For them, robots.txt is a request, not a block.
Bots pretending to be GPTBot or ClaudeBot ignore robots.txt anyway. Only verification tells who is who.

Check that it is respected

After publishing, watch for a few weeks: genuine training crawlers should stop requesting pages, while search crawlers and assistants keep coming. Knocklog shows this by agent, separates genuine agents from impostors and, on Pro, drafts a robots.txt with this policy fitted to your site.

See what your robots.txt blocks today

The free check lists every AI agent and whether it may enter your site.