Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives
Aug 4, 2025Gabriel Corral, Vaibhav Singhal, Brian Mitchell, Reid Tatoris
Summary
This post details the observed stealth crawling behavior of Perplexity, which uses undeclared user agents and rotates IP addresses and ASNs to evade robots.txt directives and WAF rules. Cloudflare has responded by de-listing Perplexity as a verified bot and implementing new heuristics in its managed rules to block this type of stealth crawling. The post also contrasts this behavior with well-behaved bots like OpenAI's ChatGPT Agent, which respects website preferences and adheres to standards like Web Bot Auth. Cloudflare's bot management system and managed rules are updated to protect customers from such evasive tactics.
More from the Website Security & Threat Management trail
- We tested our own WAF with frontier AI models. Here’s what we found
- When scanners miss the attack: how Cloudflare Client-Side Security protects storefronts
- Introducing automatic remediation policies with Cloudflare CASB
- Introducing context-aware vulnerability discovery and remediation with Cloudflare Managed Defense and OpenAI Daybreak Models
- Introducing Adaptive Intelligence: undermining the economics of every bot attack