Skip to content
CiteCheck

Technical

AI crawlers list: GPTBot, OAI-SearchBot, PerplexityBot and more

By the CiteCheck team · Updated

Short answer

AI companies use separate user agents for search, user-requested fetches and model training. For visibility in AI answers, allow the search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot, Bingbot, Applebot). Training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended) can be blocked without removing you from those search features.

The AI crawler table

Each company documents its own agents. Sorted by purpose:

User agent (robots.txt token) Company Purpose Affects AI search visibility?
OAI-SearchBot OpenAI Surfaces sites in ChatGPT search answers Yes
ChatGPT-User OpenAI Fetches a page when a ChatGPT user asks; robots.txt may not apply Indirectly
GPTBot OpenAI Crawls content that may be used to train OpenAI models No (training)
PerplexityBot Perplexity Surfaces and links sites in Perplexity results; not used for training Yes
Perplexity-User Perplexity Fetches a page when a user asks; generally ignores robots.txt Indirectly
Claude-SearchBot Anthropic Indexes content to improve Claude's search results Yes
Claude-User Anthropic Fetches pages when a Claude user asks Indirectly
ClaudeBot Anthropic Collects content that may be used for training No (training)
Googlebot Google Google Search, including AI Overviews and AI Mode Yes
Google-Extended Google Gemini training and grounding in Gemini Apps / Vertex AI; not a Search signal No for Search
Bingbot Microsoft Bing Search Yes
Applebot Apple Siri, Spotlight and Safari search; may ground Apple AI answers Yes
Applebot-Extended Apple Controls use of Applebot data for training Apple models; does not crawl No (training)

Sources: OpenAI, Perplexity, Anthropic, Google, Bing, Apple.

Three things people get wrong

1. Blocking training bots doesn't hide you from AI search. OpenAI says its settings are independent: you can allow OAI-SearchBot to appear in search results while disallowing GPTBot. Google says Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". Apple says pages that disallow Applebot-Extended can still be included in search results.

2. Blocking search bots does hide you. OpenAI says sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers, though can still appear as navigational links". Perplexity recommends allowing PerplexityBot "to ensure your site appears in search results". Anthropic says disabling Claude-SearchBot "may reduce your site's visibility and accuracy in user search results".

3. User-triggered fetchers are different. ChatGPT-User and Perplexity-User act when a person asks about a page. Both companies say robots.txt may not apply to them. If you need to block them, use your firewall.

How robots.txt matching works

Under the Robots Exclusion Protocol (RFC 9309):

  • A crawler follows the group whose User-agent line names it. If no group names it, it follows the User-agent: * group.
  • Within a group, the most specific (longest) matching rule wins. If an Allow and a Disallow rule are equally specific, Allow wins.
  • An empty Disallow: line allows everything.

Two vendor-specific details: Apple says that if robots.txt doesn't mention Applebot but does mention Googlebot, Applebot follows the Googlebot rules. Anthropic supports the non-standard Crawl-delay directive.

A common accident: a site blocks everything for * to stop scrapers, and AI search bots that aren't named inherit that block.

Copy-paste robots.txt rules

Allow AI search, opt out of training (a common middle ground):

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: *
Allow: /

Allow everything (maximum visibility, including training):

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Changes aren't instant: OpenAI cites about 24 hours for search, and Perplexity says up to 24 hours.

Check your site in seconds

The free AI crawler checker fetches your robots.txt and shows, for each agent above, whether it's allowed, blocked or partially blocked, and which group it matched. It applies the matching rules above, including Applebot's fallback to Googlebot rules.

See what AI says about your business

Free check: are you mentioned, who's named instead, which sources are cited, and what to fix.

Run a free AI visibility check

Frequently asked questions

If I block GPTBot, will I disappear from ChatGPT?
No. OpenAI says GPTBot and OAI-SearchBot are independent: disallowing GPTBot signals that content should not be used for training, while OAI-SearchBot controls whether your site appears in ChatGPT search answers.
Does blocking Google-Extended remove me from AI Overviews?
No. Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal. AI Overviews are controlled through Googlebot and snippet controls.
Do user-triggered agents obey robots.txt?
Not always. OpenAI says robots.txt rules may not apply to ChatGPT-User because actions are user-initiated, and Perplexity says Perplexity-User generally ignores robots.txt for the same reason.
How fast do robots.txt changes take effect?
OpenAI says it can take about 24 hours for search results to reflect a robots.txt update; Perplexity says up to 24 hours.

Sources

  1. OpenAI — Overview of OpenAI crawlers
  2. Perplexity — Perplexity crawlers
  3. Anthropic — Does Anthropic crawl data from the web, and how can site owners block the crawler?
  4. Google — List of Google's common crawlers
  5. Apple Support — About Applebot
  6. Bing Webmaster Tools — Which crawlers does Bing use?
  7. RFC 9309 — Robots Exclusion Protocol

More guides