robots.txt for AI Crawlers
The exact robots.txt configuration to allow GPTBot, ClaudeBot, PerplexityBot, and other AI crawlers to index your content.
Your robots.txt controls which crawlers can access your site. Most robots.txt files were written before AI crawlers existed. If yours is more than 2 years old, there is a high probability it is accidentally blocking the crawlers that determine AI citations.
Check your current robots.txt
Visit yourdomain.com/robots.txt. Look for Disallow rules affecting any of these crawlers.
Correct robots.txt configuration
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GoogleOther
Allow: /
User-agent: *
Allow: /
Sitemap: https://yourdomain.com/sitemap.xml| Crawler | Details |
|---|---|
| GPTBot | OpenAI. Powers ChatGPT training and Browse. Block = invisible to ChatGPT. |
| OAI-SearchBot | OpenAI real-time search crawler for live ChatGPT results. |
| ClaudeBot | Anthropic. Powers Claude responses. |
| PerplexityBot | Perplexity real-time citation crawler. |
| GoogleOther | Google AI Overviews — separate from Googlebot. |
What to block
Allow AI crawlers to access all public content. Block private pages only:
- •Disallow: /admin/
- •Disallow: /account/
- •Disallow: /checkout/