✦ FREE TOOL · NO SIGN-UP

AI Crawler Checker

See whether your robots.txt lets ChatGPT, Claude, Perplexity, Google and other AI crawlers fetch your pages, and which rule decides it.

Checks 13 AI search, user-requested and training crawlers. Public HTTPS sites only.

What does the AI crawler checker test?

It reads your site’s /robots.txt, finds the rules that apply to each AI crawler, and reports whether that crawler may fetch the URL you entered. It follows the robots.txt standard (RFC 9309): the group that names a crawler wins over the * group, and the longest matching rule decides.

Which AI crawlers should I allow?

If you want to appear in AI answers, allow the search and user-requested crawlers: OAI-SearchBot and ChatGPT-User for ChatGPT, Claude-SearchBot and Claude-User for Claude, PerplexityBot for Perplexity, plus Googlebot and Bingbot. Training crawlers such as GPTBot, ClaudeBot, Google-Extended and CCBot are a separate policy choice about model training.

Does blocking GPTBot remove my site from ChatGPT?

Not by itself. OpenAI documents GPTBot as its training crawler. ChatGPT search results use OAI-SearchBot, and pages a user asks ChatGPT to open are fetched by ChatGPT-User. You can block training and still allow search.

Does Google-Extended affect Google Search or AI Overviews?

No. Google states that Google-Extended does not affect inclusion or ranking in Google Search. AI Overviews and AI Mode are part of Search, so they use Googlebot and respect snippet controls such as nosnippet and max-snippet.

How do I allow an AI crawler in robots.txt?

Add a group for the crawler with an Allow rule, or remove the Disallow line that blocks it:

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Then run this check again. A firewall or bot-protection service can still block crawlers that robots.txt allows, so check those settings too.

Related free tools