TechPrufer
Get started
Vercel
Vercel logoVercel · Feature trail

AI Crawler Behavior Analysis

2 posts · Jul 2024 – Dec 2024

What this trail covers

Vercel's platform is used to analyze and understand the behavior of AI crawlers, including their scale, geographic distribution, JavaScript rendering capabilities, content type priorities, and crawling inefficiencies. This analysis helps site owners optimize their content for AI consumption and provides insights into the evolving web landscape. This post details research into Google's crawling and rendering of JavaScript-heavy sites, debunking common myths about its capabilities and providing empirical evidence on how Googlebot renders and indexes modern web applications, including those built with JavaScript frameworks and React Server Components. The research involved analyzing over 100,000 Googlebot fetches to validate Google's SEO capabilities and provide recommendations for optimizing sites for search engines.

All Vercel trails
  1. The rise of the AI crawlerThis post details Vercel's analysis of AI crawler behavior, specifically focusing on GPTBot (ChatGPT) and ClaudeBot. It quantifies their request volume (569M and 370M respectively in the past month), geographic distribution (US data centers), and critically, their lack of JavaScript rendering capabilities. The post highlights that these crawlers fetch JavaScript files but do not execute them, meaning client-side rendered content is missed. It also details content type priorities (HTML for ChatGPT, images for Claude) and significant crawling inefficiencies, with over 34% of fetches resulting in 404 errors for both crawlers. Recommendations are provided for site owners on server-side rendering, URL management, and using robots.txt or Vercel's WAF to control access. For AI users, it warns about missing JavaScript-rendered content and potential inaccuracies due to high 404 rates and inconsistent freshness.
  2. Javascript SEO: How Google Crawls, Renders & Indexes JSThis post details research conducted with MERJ to empirically analyze Googlebot's crawling and rendering capabilities for JavaScript-heavy websites. It debunks common myths about Google's ability to render client-side JavaScript, treat JS pages differently, the impact of rendering queue timing, and page discovery speed for JS-heavy sites. The methodology involved using Vercel's Edge Middleware to intercept Googlebot requests and inject a JavaScript library to track rendering completion. Findings indicate that Googlebot successfully renders 100% of analyzed HTML pages on nextjs.org, including those with complex JS, dynamic content, and streamed content via React Server Components. It also confirms that Google treats JS and non-JS pages similarly regarding status codes and meta tags. The research provides recommendations for optimizing modern web apps for SEO.