Easily manage AI crawlers with our new bot categories
Sep 29, 2023Reid Tatoris, Pawel Klimek
Summary
Introduced new Verified Bot categories (AI Crawler, Search Engine Crawler, etc.) available in the Cloudflare Rules Engine and Workers. These categories allow users to create granular firewall rules and Workers logic to manage specific types of bots. Defined criteria for 'respectful AI bots' and updated the Verified Bots policy. Announced plans for more detailed reporting on bot category traffic.
More from the Website Security & Threat Management trail
- We tested our own WAF with frontier AI models. Here’s what we found
- When scanners miss the attack: how Cloudflare Client-Side Security protects storefronts
- Introducing automatic remediation policies with Cloudflare CASB
- Introducing context-aware vulnerability discovery and remediation with Cloudflare Managed Defense and OpenAI Daybreak Models
- Introducing Adaptive Intelligence: undermining the economics of every bot attack