The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals

What this post added

This post introduces detailed analysis of AI bot crawl activity, categorizing it by purpose (training, search, user actions) and tracking the evolution of specific AI crawlers (GPTBot, ClaudeBot, Bytespider, etc.). It quantifies the 'crawl-to-refer' ratio, highlighting the imbalance between AI crawling for training and the limited referral traffic back to content creators. It also analyzes the impact of AI Overviews on referral traffic from search engines like Google.