The next step for content creators in working with AI bots: Introducing AI Crawl Control
Aug 28, 2025Will Allen, Pulkita Kini, Cam Whiteside
Summary
This post announces the general availability of AI Crawl Control, rebranding the AI Audit tool. It introduces the ability for paid Cloudflare customers to send customizable HTTP 402 ('Payment Required') status codes when blocking individual crawlers. This allows content creators to specify contact information or API details for licensing negotiations directly within the 402 response. The feature is configurable via the AI Crawl Control dashboard.
More from the Website Security & Threat Management trail
- We tested our own WAF with frontier AI models. Here’s what we found
- When scanners miss the attack: how Cloudflare Client-Side Security protects storefronts
- Introducing automatic remediation policies with Cloudflare CASB
- Introducing context-aware vulnerability discovery and remediation with Cloudflare Managed Defense and OpenAI Daybreak Models
- Introducing Adaptive Intelligence: undermining the economics of every bot attack