Free tool
Selective AI-agent blocking configuration generator
A free AI blocker for publishers. Block AI crawlers that train on your content, keep the ones that cite you and send readers back, and copy ready-to-paste config for your stack.
Choose your posture
AI training crawlers
Extract your content to train models. Blocked by the recommended posture.
GPTBot· OpenAICollects web content to train OpenAI's models.
ClaudeBot· AnthropicAnthropic's training-data crawler.
anthropic-ai· AnthropicLegacy Anthropic training user-agent.
CCBot· Common CrawlBuilds the open Common Crawl corpus used to train many models.
Bytespider· ByteDanceByteDance's AI training crawler.
Meta-ExternalAgent· MetaMeta's crawler for training AI products.
Google-Extended· GoogleOpt-out token for Gemini / Vertex AI training. Does not affect Search.
Applebot-Extended· AppleOpt-out token for Apple Intelligence training.
cohere-ai· CohereCohere's model-training crawler.
Search & citation crawlers
Can cite your content and send referral traffic. Allowed by default.
OAI-SearchBot· OpenAISurfaces and links your content in ChatGPT search results.
Claude-SearchBot· AnthropicRetrieves and cites your content in Claude answers.
PerplexityBot· PerplexityIndexes content to cite and link in Perplexity answers.
# Selective AI-agent blocking — generated with Pelcro
# Note: robots.txt is advisory. Many AI crawlers ignore it.
# Blocked — AI training crawlers
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: cohere-ai
Disallow: /
# Allowed — search / citation crawlers that send referral traffic
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
Heads up: robots.txt is advisory and a large share of AI crawlers ignore it. CDN/WAF rules enforce blocking at the edge, but user-agents can be spoofed. Request-time fingerprinting and enforcement is a planned future phase.
How to use the AI-agent blocking generator
Set your posture
Keep the recommended default — block training crawlers, allow search — or toggle any crawler on its own.
Pick your stack
Switch between robots.txt and rules for Cloudflare, Fastly, or Akamai. The output updates as you change toggles.
Copy and deploy
Click Copy, paste into your robots.txt or CDN/WAF config, and publish. No account or signup needed.
Why publishers block AI training crawlers
If you run a newspaper, magazine, or blog, you likely have no simple way to control which AI agents crawl your content. Training crawlers extract your articles to train models while sending little or no referral traffic back — so your work fuels an AI product and you see nothing in return. This free AI blocker lets you block AI crawlers selectively: shut out the ones that only take, while keeping the citation and search crawlers that can surface your content in AI answers and send readers your way.
One important caveat: robots.txt is advisory. Well-behaved crawlers honor it, but a large share of AI crawlers ignore it entirely. Edge rules at your CDN or WAF enforce blocking far more reliably, though user-agents can still be spoofed. Request-time fingerprinting and enforcement is a planned future phase — this tool generates the configuration for you today.
Would you rather your readers paid for your content?
Pelcro helps publishers put content behind paywalls, memberships, and subscriptions — so your audience pays you for the work AI crawlers try to take for free.
Explore Pelcro →How selective AI blocking works
Block the crawlers that take your content to train models, while keeping the ones that cite you and send readers back.