The rise of the AI crawler

This post details Vercel's analysis of AI crawler behavior, specifically focusing on GPTBot (ChatGPT) and ClaudeBot. It quantifies their request volume (569M and 370M respectively in the past month), geographic distribution (US data centers), and critically, their lack of JavaScript rendering capabilities. The post highlights that these crawlers fetch JavaScript files but do not execute them, meaning client-side rendered content is missed. It also details content type priorities (HTML for ChatGPT, images for Claude) and significant crawling inefficiencies, with over 34% of fetches resulting in 404 errors for both crawlers. Recommendations are provided for site owners on server-side rendering, URL management, and using robots.txt or Vercel's WAF to control access. For AI users, it warns about missing JavaScript-rendered content and potential inaccuracies due to high 404 rates and inconsistent freshness.