TechPrufer
Sign inGet started

Blogs›Vercel›AI Crawler Behavior Analysis

AI Crawler Behavior Analysis

AI Crawler Behavior Analysis

2
posts
2024

Vercel's platform is used to analyze and understand the behavior of AI crawlers, including their scale, geographic distribution, JavaScript rendering capabilities, content type priorities, and crawling inefficiencies. This analysis helps site owners optimize their content for AI consumption and provides insights into the evolving web landscape. This post details research into Google's crawling and rendering of JavaScript-heavy sites, debunking common myths about its capabilities and providing empirical evidence on how Googlebot processes modern web applications, including those built with Next.js and React Server Components. The research involved analyzing over 100,000 Googlebot fetches and implementing custom Edge Middleware to track rendering success rates and content indexing.

2024

The rise of the AI crawler

12/17/2024

This post details Vercel's data collection process and analysis of AI crawler traffic (GPTBot, ClaudeBot, etc.) across its network and on specific websites. It quantifies AI crawler request volumes, compares them to Googlebot, analyzes geographic distribution, and highlights that major AI crawlers (except Gemini and AppleBot) do not render JavaScript. It also details content type priorities (HTML, images, JS) and identifies significant crawling inefficiencies like high 404 rates and redirect following. Recommendations are provided for site owners and AI users based on these findings.

Javascript SEO: How Google Crawls, Renders & Indexes JS

7/31/2024

This post details research conducted in partnership with MERJ to empirically demystify Google's crawling and rendering of JavaScript-heavy websites. It presents findings from analyzing over 100,000 Googlebot fetches, focusing on the evolution of Google's rendering capabilities and debunking common myths. Key contributions include: 1) Demonstrating that 100% of analyzed HTML pages on nextjs.org resulted in full-page renders by Googlebot, including complex JS interactions and dynamically loaded content via API calls and RSCs. 2) Validating that Google treats JavaScript-heavy pages similarly to static HTML pages in terms of indexing and processing. 3) Providing evidence that rendering queue and timing, and page discovery speed for JS-heavy sites, are not significantly impacted as previously believed. The methodology involved custom Edge Middleware for data collection and analysis of server access logs against rendering beacons.