The crawl before the fall… of referrals: understanding AI’s impact on content providers

What this post added

Introduced a new metric on Cloudflare Radar's AI Insights page to measure the ratio of AI bot crawl requests to actual referral traffic. This metric helps content providers understand how AI bots are consuming their content versus driving traffic. The post details the methodology for calculating this ratio, including the use of User-Agent strings and Referer headers, and presents observations on crawl-to-referral ratios for various AI platforms and search engines, highlighting trends and changes over time.