Skip to content
CiteCheck

AI crawler checker

Can AI search engines read your site? Enter a domain to test its robots.txt against 13 AI crawlers and user agents from OpenAI, Perplexity, Anthropic, Google, Microsoft, Apple, Meta and Common Crawl. Blocking a search crawler such as OAI-SearchBot or PerplexityBot can keep you out of their answers.

What each crawler does and copy-paste rules: AI crawlers list and robots.txt guide.

Frequently asked questions

Which AI crawlers matter for being shown in AI answers?
The search crawlers: OAI-SearchBot for ChatGPT search, PerplexityBot for Perplexity, Claude-SearchBot for Claude, and Googlebot and Bingbot for Google's AI features and Microsoft Copilot. Training crawlers like GPTBot, ClaudeBot and Google-Extended don't control whether you're shown in search answers.
How does the checker decide if a bot is blocked?
It follows RFC 9309, the robots.txt standard: it finds the group that names the bot (or falls back to User-agent: *), then applies the longest matching Allow or Disallow rule to your homepage path. 'Partial' means the homepage is allowed but some paths are disallowed.
My robots.txt allows everything but a bot still can't reach my site. Why?
A firewall, CDN bot protection or rate limiting can block crawlers regardless of robots.txt. Check your CDN's bot settings and server logs.