If ChatGPT, Perplexity or Claude never mention your business, the cause is often a single file: robots.txt. Many sites block AI crawlers by accident, through a security plugin, a CDN setting or an old rule copied from somewhere else. This guide explains which AI crawlers exist, which ones affect whether you are cited, and exactly what to put in your robots.txt.
Quick check: run our free AI visibility checker to see in 30 seconds whether AI crawlers can reach your site.
Two kinds of AI crawler: search and training
AI companies run different bots for different jobs. The distinction matters, because blocking the wrong one can remove you from AI answers entirely.
- Search and user-triggered crawlers fetch pages so an assistant can answer a question and cite sources right now. Block these and you disappear from AI answers.
- Training crawlers collect content to train future models. Blocking them is a business decision about how your content is used. It does not, by itself, stop live AI search from citing you.
The AI crawlers that matter most
| Company | User agent | What it does |
|---|---|---|
| OpenAI | OAI-SearchBot | Indexes pages for ChatGPT search results and citations |
| OpenAI | ChatGPT-User | Fetches a page when a ChatGPT user or action asks for it |
| OpenAI | GPTBot | Collects content for model training |
| Anthropic | ClaudeBot | Collects web content for Claude |
| Anthropic | Claude-User, Claude-SearchBot | Fetch pages for user requests and Claude’s search |
| Perplexity | PerplexityBot | Indexes pages for Perplexity answers |
| Perplexity | Perplexity-User | Fetches pages when a user asks |
| Googlebot | Normal Google crawling, which also feeds AI Overviews and AI Mode | |
| Google-Extended | A control token for Gemini model training, not a separate crawler | |
| Microsoft | Bingbot | Bing search, which also powers Copilot answers |
Bot names and behaviour change, so check each company’s documentation before making a final decision.
What to put in robots.txt
For most businesses that want to be found and cited, the safest setup is to allow the search and user-triggered bots, and make a deliberate choice about training bots. A simple example:
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: ClaudeBot
Allow: /
# Optional: opt out of training
# User-agent: GPTBot
# Disallow: /
Sitemap: https://yourdomain.com/sitemap.xmlNever block Googlebot or Bingbot if you want to appear in Google AI Overviews or Copilot. Those answers are built on the normal search index.
Common mistakes that block AI by accident
- A blanket
Disallow: /left over from a staging site. - Security or bot-protection settings at the CDN or firewall level that challenge or block AI user agents, even when robots.txt allows them.
- Content that only loads with JavaScript. Many AI crawlers do not run JavaScript, so they see an empty page.
- No sitemap reference, which makes it harder for crawlers to discover your key pages.
Beyond robots.txt
Access is only the first step. To actually be cited, AI systems also need to understand who you are and find answers worth quoting. Add an llms.txt file, Organization and FAQ schema, and clear, specific answers on your key pages. Our guide on how to get cited by ChatGPT and Google AI covers the rest.
FAQs
Should I block GPTBot?
It depends on whether you are comfortable with your content being used to train models. Blocking GPTBot does not stop OAI-SearchBot from showing your pages in ChatGPT search.
Does blocking Google-Extended remove me from AI Overviews?
No. Google-Extended controls use of your content for Gemini training. AI Overviews draw on pages crawled by Googlebot.
How do I check if AI crawlers can access my site?
Run the free AI visibility audit. It reads your robots.txt the way an AI crawler does and flags any blocked bots.
