PerplexityBot: Crawl Access, IP Verification and Troubleshooting
Audit Perplexity crawl access using documented user agents, published IP ranges, WAF logs and page-level accessibility tests.
PerplexityBot is Perplexity’s automated crawler for surfacing and linking websites in search results. Perplexity says it is not used to crawl content for AI foundation-model training. To validate legitimate traffic, combine the documented user agent with Perplexity’s published IP ranges, then monitor robots, WAF and origin logs.
Perplexity-User is different: it supports user-requested page visits and generally ignores robots.txt because the user initiated the fetch.
Know the two agents
PerplexityBot
Purpose: automated search discovery and indexing. Perplexity recommends allowing it in robots.txt if you want content eligible for its search results.
Perplexity-User
Purpose: retrieve a page in response to a user action. It is not a general crawler or foundation-model training agent.
Keep both identities distinct in logs and policy.
Configure robots.txt deliberately
A basic allow rule is:
```txt
User-agent: PerplexityBot
Allow: /
```
Robots permission does not override authentication, WAF blocks or server errors, and it does not guarantee inclusion. Perplexity says changes may take up to 24 hours to reflect.
Review the broader robots.txt guide for AI crawlers before changing global rules.
Verify IP addresses
User-agent strings can be spoofed. Perplexity publishes separate JSON endpoints for PerplexityBot and Perplexity-User IP ranges and recommends combining user-agent matching with IP verification.
Build an automated process that:
- fetches the official JSON endpoints;
- validates the response;
- updates WAF IP sets safely;
- logs the version and timestamp;
- keeps the last known-good set if retrieval fails.
Do not hardcode a copied list indefinitely because published ranges can change.
Check the full request path
Test:
- CDN and bot-management decisions;
- edge redirects;
- robots response;
- origin status;
- TLS and DNS;
- HTML payload;
- canonical URL;
- rate limiting;
- geography-specific rules.
A 200 returned by the CDN does not prove the bot received the real article; it might receive a challenge or app shell.
Use logs for diagnosis
Record timestamp, IP, user agent, URL, status, bytes, response time and edge action. Separate successful HTML fetches from images and assets.
Common patterns:
- repeated 403: WAF or access rule;
- 429: rate limit;
- 301 loop: canonical or host logic;
- 200 with tiny payload: challenge or empty shell;
- 5xx: origin or upstream failure;
- no requests: discovery, linking or DNS issue.
Test representative templates
Check the homepage, category, article, product and location templates. Confirm important text appears in the response without a browser session.
Use the AI crawler audit for a multi-agent inventory and How to Get Cited by Perplexity for the editorial and authority layer.
Maintain the configuration
Review official documentation quarterly and after WAF, CDN or platform changes. Alert on sudden changes in allowed fetches, blocks and status codes.
Do not whitelist every request merely because it contains “Perplexity.” Match the official IP data.
Frequently asked questions
Is PerplexityBot used for training foundation models?
Perplexity’s documentation says it is not.
Does Perplexity-User obey robots.txt?
Perplexity says it generally ignores robots.txt because it handles a user-requested fetch.
Will allowing PerplexityBot guarantee citations?
No. It removes one possible access barrier; retrieval and selection remain separate.
