The crawl before the fall… of referrals: understanding AI’s impact on content providers
This post introduces a new metric, the 'crawl-to-refer ratio', displayed on Cloudflare Radar's AI Insights page. This metric quantifies the relationship between AI bot crawl requests for HTML content and the number of HTML page requests referred by AI platforms. The measurement methodology involves dividing HTML page requests from relevant user agents by HTML page requests where the Referer header contains a hostname associated with a given search or AI platform. The post presents observational data on these ratios for various AI platforms and search engines, analyzes trends in crawling and referral traffic, and discusses the implications for content providers. It also references existing tools for blocking AI crawlers and new initiatives for fair value exchange.
More from the Website Security & Threat Management trail
- We tested our own WAF with frontier AI models. Here’s what we found
- When scanners miss the attack: how Cloudflare Client-Side Security protects storefronts
- Introducing automatic remediation policies with Cloudflare CASB
- Introducing context-aware vulnerability discovery and remediation with Cloudflare Managed Defense and OpenAI Daybreak Models
- Introducing Adaptive Intelligence: undermining the economics of every bot attack