# AI crawlers list: GPTBot, OAI-SearchBot, PerplexityBot and more

Source: https://citecheck.torontosoft.com/guides/ai-crawlers-robots-txt
Updated: 2026-11-20

> AI companies use separate user agents for search, user-requested fetches and model training. For visibility in AI answers, allow the search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot, Bingbot, Applebot). Training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended) can be blocked without removing you from those search features.

## The AI crawler table

Each company documents its own agents. Sorted by purpose:

| User agent (robots.txt token) | Company | Purpose | Affects AI search visibility? |
|---|---|---|---|
| `OAI-SearchBot` | OpenAI | Surfaces sites in ChatGPT search answers | **Yes** |
| `ChatGPT-User` | OpenAI | Fetches a page when a ChatGPT user asks; robots.txt may not apply | Indirectly |
| `GPTBot` | OpenAI | Crawls content that may be used to train OpenAI models | No (training) |
| `PerplexityBot` | Perplexity | Surfaces and links sites in Perplexity results; not used for training | **Yes** |
| `Perplexity-User` | Perplexity | Fetches a page when a user asks; generally ignores robots.txt | Indirectly |
| `Claude-SearchBot` | Anthropic | Indexes content to improve Claude's search results | **Yes** |
| `Claude-User` | Anthropic | Fetches pages when a Claude user asks | Indirectly |
| `ClaudeBot` | Anthropic | Collects content that may be used for training | No (training) |
| `Googlebot` | Google | Google Search, including AI Overviews and AI Mode | **Yes** |
| `Google-Extended` | Google | Gemini training and grounding in Gemini Apps / Vertex AI; not a Search signal | No for Search |
| `Bingbot` | Microsoft | Bing Search | **Yes** |
| `Applebot` | Apple | Siri, Spotlight and Safari search; may ground Apple AI answers | **Yes** |
| `Applebot-Extended` | Apple | Controls use of Applebot data for training Apple models; does not crawl | No (training) |

Sources: [OpenAI](https://platform.openai.com/docs/bots), [Perplexity](https://docs.perplexity.ai/guides/bots), [Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), [Google](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers), [Bing](https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0), [Apple](https://support.apple.com/en-us/119829).

## Three things people get wrong

**1. Blocking training bots doesn't hide you from AI search.** OpenAI says its settings are independent: you can allow OAI-SearchBot to appear in search results while disallowing GPTBot. Google says Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". Apple says pages that disallow Applebot-Extended can still be included in search results.

**2. Blocking search bots does hide you.** OpenAI says sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers, though can still appear as navigational links". Perplexity recommends allowing PerplexityBot "to ensure your site appears in search results". Anthropic says disabling Claude-SearchBot "may reduce your site's visibility and accuracy in user search results".

**3. User-triggered fetchers are different.** ChatGPT-User and Perplexity-User act when a person asks about a page. Both companies say robots.txt may not apply to them. If you need to block them, use your firewall.

## How robots.txt matching works

Under the Robots Exclusion Protocol ([RFC 9309](https://www.rfc-editor.org/rfc/rfc9309)):

- A crawler follows the group whose `User-agent` line **names it**. If no group names it, it follows the `User-agent: *` group.
- Within a group, the **most specific (longest) matching rule** wins. If an `Allow` and a `Disallow` rule are equally specific, `Allow` wins.
- An empty `Disallow:` line allows everything.

Two vendor-specific details: Apple says that if robots.txt doesn't mention Applebot but does mention Googlebot, **Applebot follows the Googlebot rules**. Anthropic supports the non-standard `Crawl-delay` directive.

A common accident: a site blocks everything for `*` to stop scrapers, and AI search bots that aren't named inherit that block.

## Copy-paste robots.txt rules

**Allow AI search, opt out of training (a common middle ground):**

```
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: *
Allow: /
```

**Allow everything (maximum visibility, including training):**

```
User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml
```

Changes aren't instant: OpenAI cites about 24 hours for search, and Perplexity says up to 24 hours.

## Check your site in seconds

The free [AI crawler checker](/tools/ai-crawler-checker) fetches your robots.txt and shows, for each agent above, whether it's allowed, blocked or partially blocked, and which group it matched. It applies the matching rules above, including Applebot's fallback to Googlebot rules.

## FAQ

### If I block GPTBot, will I disappear from ChatGPT?

No. OpenAI says GPTBot and OAI-SearchBot are independent: disallowing GPTBot signals that content should not be used for training, while OAI-SearchBot controls whether your site appears in ChatGPT search answers.

### Does blocking Google-Extended remove me from AI Overviews?

No. Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal. AI Overviews are controlled through Googlebot and snippet controls.

### Do user-triggered agents obey robots.txt?

Not always. OpenAI says robots.txt rules may not apply to ChatGPT-User because actions are user-initiated, and Perplexity says Perplexity-User generally ignores robots.txt for the same reason.

### How fast do robots.txt changes take effect?

OpenAI says it can take about 24 hours for search results to reflect a robots.txt update; Perplexity says up to 24 hours.

## Sources

1. [OpenAI — Overview of OpenAI crawlers](https://platform.openai.com/docs/bots)
2. [Perplexity — Perplexity crawlers](https://docs.perplexity.ai/guides/bots)
3. [Anthropic — Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
4. [Google — List of Google's common crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers)
5. [Apple Support — About Applebot](https://support.apple.com/en-us/119829)
6. [Bing Webmaster Tools — Which crawlers does Bing use?](https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0)
7. [RFC 9309 — Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309)
