Reference · AI crawlers
Every major AI crawler, what it actually controls, and how to allow or block it. Each fact is checked against the vendor’s own documentation and carries a last-verified date. The one rule most guides get wrong: the crawlers that get you cited are not the same as the ones that train a model.
Discovery / citation crawler, decides whether you can appear and be cited in a live AI answer or AI search result. Allow these to be eligible.
OpenAI · ChatGPT search
OpenAI's crawler for surfacing websites in ChatGPT's search results. Allow it to be eligible for citations and links in ChatGPT search.
documented · verified 2026-09-04
Google · Google Search + AI Overviews
Google's search crawler. It powers Google Search, and AI Overviews / AI Mode draw from the same Search index, so Googlebot access gates both.
documented · verified 2026-09-04
Perplexity · Perplexity search
Perplexity's search crawler. Allow it to have the best chance of appearing and being cited in Perplexity answers.
documented · verified 2026-09-04
Anthropic · Claude search
Anthropic's search crawler. It indexes content to improve Claude's search results, allow it to be eligible for citations in Claude search.
documented · verified 2026-09-04
Microsoft · Bing Search + Copilot
Microsoft's Bing search crawler. Microsoft Copilot is grounded in Bing web results, so Bingbot access affects both Bing and Copilot.
observed · verified 2026-09-04
User-initiated fetcher, visits a page only when a person asks the assistant about it. Often ignores robots.txt because it acts on a direct user request.
OpenAI · ChatGPT (user-initiated)
OpenAI's user-initiated fetcher. It visits a page only when a person asks ChatGPT about it, not an automatic crawler.
documented · verified 2026-09-04
Perplexity · Perplexity (user-initiated)
Perplexity's user-initiated fetcher. It visits a page when a person asks Perplexity about it, and generally disregards robots.txt because it acts on a user request.
documented · verified 2026-09-04
Anthropic · Claude (user-initiated)
Anthropic's user-initiated fetcher. It visits a page when a person asks Claude about it.
documented · verified 2026-09-04
Training crawler, gathers content to train or ground a model. Blocking one opts you out of training; it does NOT remove you from live AI answers.
OpenAI · OpenAI model training
OpenAI's model-training crawler. Blocking it opts your content out of training, it does NOT remove you from ChatGPT search answers.
documented · verified 2026-09-04
Google · Gemini training + grounding
A Google control token for Gemini training and grounding. Google states it does NOT affect inclusion or ranking in Google Search.
documented · verified 2026-09-04
Anthropic · Anthropic model training
Anthropic's training crawler. Blocking it opts your content out of training, it does NOT remove you from Claude's search answers.
documented · verified 2026-09-04
Common Crawl · Common Crawl open corpus
The crawler for Common Crawl's open web archive. Its data is a widely used training corpus, so blocking it is a training choice, not a live-answer one.
documented · verified 2026-09-04
An AI engine will only cite a page it can reach. But “allow the AI crawlers” is too blunt, the crawlers split into three jobs, and confusing them is the most common way sites accidentally opt out of citations or waste effort blocking the wrong bot.
The discovery / citation crawlers: OAI-SearchBot (ChatGPT), Googlebot (Google Search and AI Overviews), PerplexityBot (Perplexity), Claude-SearchBot (Claude), and Bingbot (Bing and Copilot). Blocking any of these makes you ineligible to appear in that engine’s live answers.
No. GPTBot and Google-Extended are training controls. Blocking them opts your content out of model training; it does not remove you from live AI answers. Google states plainly that Google-Extended does not affect inclusion in Google Search.
An agent like ChatGPT-User, Perplexity-User or Claude-User that visits a page only when a person asks the assistant about it. These often disregard robots.txt because they act on a direct user request rather than crawling in the background.
Every entry carries a "last verified" date and a link to the vendor’s own documentation. This registry was last verified on 2026-09-04. The machine-readable version is at /data/ai-crawlers.json.
The next question
Being crawlable is step one. The AI Visibility Checker shows whether ChatGPT, Google AI Overviews, Perplexity and Gemini name and cite your brand, and who they name instead.
Keep going