From Googlebot to GPTBot: Who’s crawling your site in 2025
Jul 1, 2025João Tomé, Jorge Pacheco, Carlos Azevedo
Summary
This post analyzes recent trends in web crawling, focusing on the emergence and growth of AI crawlers (e.g., GPTBot, Meta-ExternalAgent) and traditional search crawlers (e.g., Googlebot). It quantises the growth of AI and search crawler traffic by 18% year-over-year, with Googlebot showing a 96% increase. The post details the top AI crawlers, their purposes, and the challenges they pose regarding content rights and infrastructure. It also discusses how webmasters can manage AI crawlers using robots.txt and Cloudflare's AI Audit tools. The analysis highlights shifts in the AI crawler landscape, with OpenAI and Meta becoming dominant players.
More from the Website Security & Threat Management trail
- We tested our own WAF with frontier AI models. Here’s what we found
- When scanners miss the attack: how Cloudflare Client-Side Security protects storefronts
- Introducing automatic remediation policies with Cloudflare CASB
- Introducing context-aware vulnerability discovery and remediation with Cloudflare Managed Defense and OpenAI Daybreak Models
- Introducing Adaptive Intelligence: undermining the economics of every bot attack