The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals

This post analyzes the shift in AI bot traffic patterns, noting a significant increase in training-related crawling and a corresponding decrease in referral traffic from search engines to news sites. It quantifies the growth of specific AI crawlers (GPTBot, ClaudeBot, Meta-ExternalAgent) and the decline of others (Amazonbot, Bytespider). The analysis introduces the 'crawl-to-refer ratio' to illustrate the imbalance between bot activity and user referrals, providing specific metrics for various AI platforms. The post also details the growth of AI-only crawlers and the overall trend of AI and search crawling, including seasonal fluctuations.