Robots.txt for AI Crawlers: Safe Configuration Examples
Understand the major AI crawler user agents and configure robots.txt deliberately for search visibility, training controls and user-requested retrieval.
The safest way to configure robots.txt for AI crawlers is to decide separately whether you want search discovery, user-requested retrieval and model-training access. Then express those decisions with provider-specific rules and verify that your CDN, firewall and page-level directives behave consistently.
Copying a universal “block AI” or “allow AI” file is risky because providers use different agents for different purposes. Robots permission also does not guarantee indexing, citation or referral traffic.
A Safe Starting Pattern
The following example permits documented search crawlers while blocking OpenAI's documented training crawler. It is an illustration, not a universal policy:
```
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: *
Disallow: /admin/
Disallow: /account/
Sitemap: https://www.example.com/sitemap.xml
```
OpenAI says OAI-SearchBot controls eligibility for summaries and snippets in ChatGPT search, while GPTBot controls potential use for model training. Perplexity says PerplexityBot surfaces and links sites in search results and is not a foundation-model training crawler. Review the official documentation at implementation time because policies and agent names can change.
Separate the Three Access Purposes
| Purpose | Representative agent | What the rule controls |
|---|---|---|
| Search discovery | OAI-SearchBot, PerplexityBot | Whether the provider's crawler may fetch eligible public content for search |
| User-requested retrieval | ChatGPT-User, Perplexity-User | Fetches initiated in response to user actions; documented behavior can differ from normal crawling |
| Potential training | GPTBot | Whether OpenAI may crawl the content for potential model improvement |
Do not infer one provider's taxonomy applies to another. Maintain a policy register with a first-party documentation link for every agent you configure.
Configuration 1: Allow Search, Block Training
Use this pattern when leadership wants discoverability in ChatGPT search but has chosen not to permit GPTBot training access:
```
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
```
OpenAI documents these controls as independent. Blocking GPTBot does not require blocking OAI-SearchBot.
Configuration 2: Allow Only a Public Knowledge Area
```
User-agent: OAI-SearchBot
Disallow: /
Allow: /help/
Allow: /research/
User-agent: PerplexityBot
Disallow: /
Allow: /help/
Allow: /research/
```
Use narrow rules carefully. Test them with the same path-matching semantics the provider follows. Ensure public pages are internally linked and included in a sitemap; permission alone does not create discovery.
Configuration 3: Block Sensitive Application Paths for Everyone
```
User-agent: *
Disallow: /admin/
Disallow: /account/
Disallow: /checkout/
Disallow: /internal-search/
```
This reduces crawling, but robots.txt is public and is not a security control. Protect sensitive resources with authentication and authorization. Do not list secret paths merely to hide them.
Configuration 4: Block OpenAI Search Entirely
```
User-agent: OAI-SearchBot
Disallow: /
```
OpenAI notes that a disallowed page discovered through another source may still appear as a title and link in some circumstances. If the page must not appear in search results, use an appropriate noindex directive and allow the crawler to read it. Confirm the current provider guidance before relying on this behavior.
Understand Rule Conflicts
Review the complete file, not an isolated group. Common problems include:
- A wildcard block that teams assume a specific allow overrides.
- Different robots files on
www, apex, subdomains or protocols. - Staging rules deployed to production.
- Path casing and encoded URL differences.
- Rules aimed at parameters that do not match the real URL.
- An old plugin regenerating the file after deployment.
Retrieve the production URL and save the exact response as audit evidence.
Robots.txt Does Not Control the Firewall
A crawler can be allowed in robots.txt and still receive a 403, 429 or JavaScript challenge at the edge. Perplexity publishes IP-range files for its agents, and OpenAI links to crawler identification information from its publisher documentation.
Where IP verification is available:
- Fetch ranges only from the provider's official endpoint.
- Validate source IP and user agent together.
- Automate safe updates to changing ranges.
- Log allow, challenge and deny decisions.
- Rate-limit proportionately instead of creating a permanent blanket block by accident.
The AI crawler audit covers the full edge-to-origin test.
How to Verify a Change
After deployment:
- Fetch
/robots.txtfrom every relevant host. - Confirm status, content type and cache freshness.
- Test allowed and disallowed representative URLs.
- Inspect WAF and origin logs.
- Check meta robots,
X-Robots-Tagand canonicals. - Confirm important content appears in the returned HTML.
- Monitor crawl attempts over the following weeks.
Do not submit repeated prompts to an AI platform and treat a missing citation as proof the robots rule failed. Citation depends on more than access.
Governance Checklist
- Legal or data-governance owner approves training policy.
- Search owner approves discovery policy.
- Security reviews user-triggered access and WAF controls.
- Engineering owns deployment and monitoring.
- Every rule has a purpose and review date.
- Private data is protected independently of robots rules.
- The change is tested against Googlebot and Bingbot to prevent collateral damage.
Frequently Asked Questions
Can I Allow OAI-SearchBot and Block GPTBot?
Yes. OpenAI documents separate controls for search visibility and potential training. Use the exact current agent names from its publisher guidance.
Does Allowing PerplexityBot Guarantee a Citation?
No. Perplexity recommends allowing the agent for search appearance, but retrieval permission does not guarantee source selection or attribution.
Should I Block Every Unknown AI User Agent?
That is a policy and security decision, not an SEO default. Unknown agents may be spoofed. Protect resources based on sensitivity and verified identity rather than an unmaintained name list.
