Technical
AI crawlers list: GPTBot, OAI-SearchBot, PerplexityBot and more
By the CiteCheck team · Updated
Short answer
AI companies use separate user agents for search, user-requested fetches and model training. For visibility in AI answers, allow the search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot, Bingbot, Applebot). Training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended) can be blocked without removing you from those search features.
The AI crawler table
Each company documents its own agents. Sorted by purpose:
| User agent (robots.txt token) | Company | Purpose | Affects AI search visibility? |
|---|---|---|---|
OAI-SearchBot |
OpenAI | Surfaces sites in ChatGPT search answers | Yes |
ChatGPT-User |
OpenAI | Fetches a page when a ChatGPT user asks; robots.txt may not apply | Indirectly |
GPTBot |
OpenAI | Crawls content that may be used to train OpenAI models | No (training) |
PerplexityBot |
Perplexity | Surfaces and links sites in Perplexity results; not used for training | Yes |
Perplexity-User |
Perplexity | Fetches a page when a user asks; generally ignores robots.txt | Indirectly |
Claude-SearchBot |
Anthropic | Indexes content to improve Claude's search results | Yes |
Claude-User |
Anthropic | Fetches pages when a Claude user asks | Indirectly |
ClaudeBot |
Anthropic | Collects content that may be used for training | No (training) |
Googlebot |
Google Search, including AI Overviews and AI Mode | Yes | |
Google-Extended |
Gemini training and grounding in Gemini Apps / Vertex AI; not a Search signal | No for Search | |
Bingbot |
Microsoft | Bing Search | Yes |
Applebot |
Apple | Siri, Spotlight and Safari search; may ground Apple AI answers | Yes |
Applebot-Extended |
Apple | Controls use of Applebot data for training Apple models; does not crawl | No (training) |
Sources: OpenAI, Perplexity, Anthropic, Google, Bing, Apple.
Three things people get wrong
1. Blocking training bots doesn't hide you from AI search. OpenAI says its settings are independent: you can allow OAI-SearchBot to appear in search results while disallowing GPTBot. Google says Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". Apple says pages that disallow Applebot-Extended can still be included in search results.
2. Blocking search bots does hide you. OpenAI says sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers, though can still appear as navigational links". Perplexity recommends allowing PerplexityBot "to ensure your site appears in search results". Anthropic says disabling Claude-SearchBot "may reduce your site's visibility and accuracy in user search results".
3. User-triggered fetchers are different. ChatGPT-User and Perplexity-User act when a person asks about a page. Both companies say robots.txt may not apply to them. If you need to block them, use your firewall.
How robots.txt matching works
Under the Robots Exclusion Protocol (RFC 9309):
- A crawler follows the group whose
User-agentline names it. If no group names it, it follows theUser-agent: *group. - Within a group, the most specific (longest) matching rule wins. If an
Allowand aDisallowrule are equally specific,Allowwins. - An empty
Disallow:line allows everything.
Two vendor-specific details: Apple says that if robots.txt doesn't mention Applebot but does mention Googlebot, Applebot follows the Googlebot rules. Anthropic supports the non-standard Crawl-delay directive.
A common accident: a site blocks everything for * to stop scrapers, and AI search bots that aren't named inherit that block.
Copy-paste robots.txt rules
Allow AI search, opt out of training (a common middle ground):
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: *
Allow: /
Allow everything (maximum visibility, including training):
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xml
Changes aren't instant: OpenAI cites about 24 hours for search, and Perplexity says up to 24 hours.
Check your site in seconds
The free AI crawler checker fetches your robots.txt and shows, for each agent above, whether it's allowed, blocked or partially blocked, and which group it matched. It applies the matching rules above, including Applebot's fallback to Googlebot rules.
See what AI says about your business
Free check: are you mentioned, who's named instead, which sources are cited, and what to fix.
Run a free AI visibility checkFrequently asked questions
- If I block GPTBot, will I disappear from ChatGPT?
- No. OpenAI says GPTBot and OAI-SearchBot are independent: disallowing GPTBot signals that content should not be used for training, while OAI-SearchBot controls whether your site appears in ChatGPT search answers.
- Does blocking Google-Extended remove me from AI Overviews?
- No. Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal. AI Overviews are controlled through Googlebot and snippet controls.
- Do user-triggered agents obey robots.txt?
- Not always. OpenAI says robots.txt rules may not apply to ChatGPT-User because actions are user-initiated, and Perplexity says Perplexity-User generally ignores robots.txt for the same reason.
- How fast do robots.txt changes take effect?
- OpenAI says it can take about 24 hours for search results to reflect a robots.txt update; Perplexity says up to 24 hours.
Sources
- OpenAI — Overview of OpenAI crawlers
- Perplexity — Perplexity crawlers
- Anthropic — Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Google — List of Google's common crawlers
- Apple Support — About Applebot
- Bing Webmaster Tools — Which crawlers does Bing use?
- RFC 9309 — Robots Exclusion Protocol
More guides
Optimize
Google AI Overviews vs SEO: what changes and what doesn't
What Google says about ranking in AI Overviews and AI Mode, what Pew's click data shows, and how to adjust an SEO plan without chasing myths.
Compare
AI visibility tools compared: Otterly, Peec, Profound, Semrush
Honest comparison of AI visibility trackers OtterlyAI, Peec AI, Profound and Semrush: published prices, prompt limits and engines, checked Nov 2026.
Optimize
GEO checklist: 20 steps to get cited by AI search
A practical GEO (generative engine optimization) checklist: crawler access, content, structured data, third-party listings and measurement, with sources.