AI crawlers / Claude-User
Claude-User
Anthropic · Claude (user-initiated) · User-initiated fetcher
Anthropic's user-initiated fetcher. It visits a page when a person asks Claude about it.
At a glance
Affects live AI answersIndirect
Affects search inclusion,
Used for model trainingNo
robots.txt tokenClaude-User
Source confidencedocumented · verified 2026-09-04
Usually allowed, it fetches only when a person asks the assistant about your page.
What Claude-User is
Claude-User supports Claude users: when individuals ask Claude questions, it may access websites using a Claude-User agent to answer them.
It fetches on demand in response to a user's request, rather than crawling in the background, and does not gather training data.
Allowing Claude-User lets Claude read a page a user points it at; blocking it prevents that on-demand fetch.
Controlling Claude-User in robots.txt
Claude-User acts on a direct user request and, per Anthropic, generally does not treat robots.txt as binding the way a background crawler does. The rules below express your intent, but a user-initiated fetch may still occur when a person explicitly points the assistant at your page.
Allow Claude-User:
User-agent: Claude-User
Allow: /Block Claude-User:
User-agent: Claude-User
Disallow: /Check what your live site actually allows with the Robots.txt + AI-Bot Tester.
Source
Verified on 2026-09-04 against Anthropic, Does Anthropic crawl the web?. Confidence: documented, stated in the vendor’s own documentation.
The next question
Your page is AI-ready. But does AI actually cite you?
Being crawlable is step one. The AI Visibility Checker shows whether ChatGPT, Google AI Overviews, Perplexity and Gemini name and cite your brand, and who they name instead.
Keep going