Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives

What this post added

This post details the observed stealth crawling behavior of Perplexity, an AI-powered answer engine, which attempts to evade website no-crawl directives by obscuring its identity. Cloudflare observed Perplexity modifying its user agent, changing its source ASNs, and ignoring robots.txt files. Cloudflare has de-listed Perplexity as a verified bot and added heuristics to its managed rules to block this stealth crawling. The post also contrasts this behavior with well-meaning bot operators like OpenAI, who adhere to transparency and respect website preferences. Cloudflare outlines how customers can protect themselves using bot management systems and managed rules.