AI crawlers / ClaudeBot
ClaudeBot
Anthropic · Anthropic model training · Training crawler
Anthropic's training crawler. Blocking it opts your content out of training, it does NOT remove you from Claude's search answers.
At a glance
Affects live AI answersNo
Affects search inclusion,
Used for model trainingYes
robots.txt tokenClaudeBot
Source confidencedocumented · verified 2026-09-04
Your choice, this is a training control. Blocking it does not remove you from live AI answers.
What ClaudeBot is
ClaudeBot is the crawler Anthropic uses to collect web content that could contribute to training its models. Anthropic states it "helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training."
ClaudeBot is a training control. Blocking it does not remove you from Claude's live search answers, that is governed by Claude-SearchBot.
Treating ClaudeBot as Anthropic's citation crawler is a common error; the citation-relevant agent is Claude-SearchBot.
Controlling ClaudeBot in robots.txt
Add the matching group to the robots.txt at your site root.
Allow ClaudeBot:
User-agent: ClaudeBot
Allow: /Block ClaudeBot:
User-agent: ClaudeBot
Disallow: /Check what your live site actually allows with the Robots.txt + AI-Bot Tester.
Source
Verified on 2026-09-04 against Anthropic, Does Anthropic crawl the web?. Confidence: documented, stated in the vendor’s own documentation.
The next question
Your page is AI-ready. But does AI actually cite you?
Being crawlable is step one. The AI Visibility Checker shows whether ChatGPT, Google AI Overviews, Perplexity and Gemini name and cite your brand, and who they name instead.
Keep going