AI crawlers / Googlebot
Googlebot
Google · Google Search + AI Overviews · Discovery / citation crawler
Google's search crawler. It powers Google Search, and AI Overviews / AI Mode draw from the same Search index, so Googlebot access gates both.
At a glance
Affects live AI answersYes
Affects search inclusionYes
Used for model trainingNo
robots.txt tokenGooglebot
Source confidencedocumented · verified 2026-09-04
Allow it, this crawler decides whether you can be cited in this engine’s answers.
What Googlebot is
Googlebot is the crawler for Google Search, it crawls and indexes pages for ranking. Google's AI Overviews and AI Mode are part of Search and draw on the Google Search index rather than a separate crawler, so Googlebot access is what makes a page eligible to be surfaced there.
Blocking Googlebot removes you from Google Search and, by extension, from AI Overviews. There is no separate crawler you allow to appear in AI Overviews.
Do not confuse Googlebot with Google-Extended. Google-Extended is a training/grounding control for Gemini and, per Google, does not affect Google Search inclusion at all.
Controlling Googlebot in robots.txt
Add the matching group to the robots.txt at your site root.
Allow Googlebot:
User-agent: Googlebot
Allow: /Block Googlebot:
User-agent: Googlebot
Disallow: /Check what your live site actually allows with the Robots.txt + AI-Bot Tester.
Source
Verified on 2026-09-04 against Google, Common crawlers. Confidence: documented, stated in the vendor’s own documentation.
The next question
Your page is AI-ready. But does AI actually cite you?
Being crawlable is step one. The AI Visibility Checker shows whether ChatGPT, Google AI Overviews, Perplexity and Gemini name and cite your brand, and who they name instead.
Keep going