Google’s AI advantage: why crawler separation is the only path to a fair Internet

What this post added

This post analyzes the competitive advantage Google gains from its dual-purpose Googlebot, which crawls content for both search indexing and generative AI training. It presents Cloudflare data showing Googlebot accesses significantly more unique URLs than other AI crawlers and highlights that publishers are less likely to block Googlebot due to its search traffic referral importance. The post argues that this asymmetry disadvantages other AI companies and advocates for stronger publisher controls over content usage for AI, critiquing the CMA's proposed remedies as insufficient.