A deeper look at AI crawlers: breaking down traffic by purpose and industry

This post introduces new capabilities on Cloudflare Radar to analyze AI crawler traffic. It adds 'Crawl purpose' (Training, Search, User action, Undeclared) and 'Industry set' filtering to the AI Insights page. This allows users to break down AI bot traffic by its intended use and compare it against industry benchmarks. The post provides examples of how these new filters reveal different crawling patterns and crawl-to-refer ratios across various industries (News and Publications, Computer and Electronics) and specific use cases (Cryptocurrency).