Find out in seconds whether ChatGPT, Claude, Perplexity, Google AI Overviews, Gemini and Copilot are allowed to crawl your website. Enter a domain and we read its robots.txt the way each AI crawler does. You get the result for every bot, the rule that applies, and whether you have an llms.txt file.
Free, no sign-up. Checks 19 AI and search crawlers against your robots.txt, plus llms.txt.
What this AI crawler checker tests
AI assistants can only cite pages their crawlers are allowed to read. Many sites block them by accident, through a security plugin, a CDN bot-protection setting or a copied robots.txt rule. This free tool checks your live robots.txt against the 19 crawlers that matter most for AI search and AI training. For each one it shows:
- Allowed or blocked for your homepage, or for any page path you enter, such as yoursite.com/blog/
- The exact rule that applies, whether that is a bot-specific group or the catch-all
User-agent: *group - Whether you publish an llms.txt file, a short guide that helps AI models understand your site
Crawler access is only step one. To see whether AI can also understand and cite your content (schema, titles, FAQs, facts, trust signals and speed), run the free AI Visibility Audit, which runs 20 checks.
AI crawlers: search bots vs training bots
Not every AI bot does the same job, so you don’t have to treat them all the same.
| Crawler | Company | What it does | Block it? |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Finds pages to show and cite in ChatGPT search | Keep allowed if you want ChatGPT traffic |
| ChatGPT-User | OpenAI | Opens a page when a user asks ChatGPT about it | Keep allowed |
| GPTBot | OpenAI | Collects content for model training | Your choice |
| Claude-SearchBot / Claude-User | Anthropic | Search results and user-requested fetches in Claude | Keep allowed |
| ClaudeBot | Anthropic | Model training | Your choice |
| PerplexityBot / Perplexity-User | Perplexity | Indexes and fetches pages for Perplexity answers | Keep allowed |
| Googlebot | Google Search, including AI Overviews and AI Mode | Never block | |
| Google-Extended | Controls use of your content for Gemini training and grounding. It doesn’t affect Search ranking. | Your choice | |
| Bingbot | Microsoft | Bing search, which also feeds Copilot and other assistants | Never block |
| Applebot-Extended | Apple | Apple Intelligence training | Your choice |
| CCBot | Common Crawl | Open web dataset used to train many models | Your choice |
A common setup is to allow the search and user bots, so you can be found and cited, and to decide separately on the training bots.
How to allow AI crawlers in robots.txt
If the checker shows a search bot as blocked, find the rule it names and remove it, or add an explicit allow group. This example lets the AI answer engines in while opting out of training:
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yoursite.com/sitemap_index.xmlIf robots.txt looks fine but AI tools still can’t read your pages, check your CDN or firewall. Cloudflare and similar services have “block AI bots” switches that work separately from robots.txt.
What is llms.txt?
llms.txt is a short markdown file at the root of your site, such as yoursite.com/llms.txt. It tells AI models what your site is about and links to your most important pages. It is a proposed standard rather than a requirement, but it takes ten minutes to add and makes your key pages easier for AI tools to pick up. See ours as an example.
FAQs
How do I check if my website blocks ChatGPT?
Enter your domain in the checker above. If OAI-SearchBot or ChatGPT-User shows as blocked, ChatGPT can’t read or cite your pages in its answers. GPTBot only controls training.
Does blocking GPTBot remove my site from ChatGPT search?
No. OpenAI uses OAI-SearchBot for ChatGPT search and GPTBot for training, and each follows its own robots.txt rules. You can block GPTBot and still appear in ChatGPT search.
Does blocking Google-Extended hurt my Google rankings or AI Overviews?
Google says Google-Extended doesn’t affect ranking in Google Search. AI Overviews are built from the normal Googlebot index, so keep Googlebot allowed.
Why does the tool say it couldn’t read my robots.txt?
Your server or CDN returned an error, or blocked our request. If bot protection is blocking a simple check like this one, it may be blocking AI crawlers too, so review your firewall and bot settings.
Is this AI crawler checker free?
Yes. It is free with no sign-up. For a full report on how AI sees your site, try the free AI Visibility Audit.