Easily manage AI crawlers with our new bot categories

What this post added

Introduced new Verified Bot categories (Search Engine Crawler, Aggregator, AI Crawler, Page Preview, Advertising, Academic Research, Accessibility, Feed Fetcher, Security, Webhooks) available in the Cloudflare Rules Engine and Workers. This allows for more granular control over bot traffic, enabling specific responses to different bot types, such as blocking AI crawlers while allowing search engine crawlers. Also detailed criteria for tagging respectful AI bots, expanding the Verified Bots policy.