From Googlebot to GPTBot: Who’s crawling your site in 2025

What this post added

This post analyzes the evolving landscape of web crawlers, with a specific focus on the rise of AI crawlers. It presents data on the changing market share of AI bots like GPTBot, ClaudeBot, and Meta-ExternalAgent, and highlights the significant growth of Googlebot. The post also discusses the implications of AI crawling for content creators and the tools Cloudflare is developing to manage and enforce crawler policies, such as AI Audit.