Introducing pay per crawl: Enabling content owners to charge AI crawlers for access
Jul 1, 2025Will Allen, Simon Newton
Summary
Introduces 'Pay per crawl', a new feature enabling content owners to charge AI crawlers for access. This leverages HTTP response code 402, HTTP Message Signatures for crawler authentication and payment intent, and Cloudflare's role as Merchant of Record. Publishers can define pricing and control access (Allow, Charge, Block) via a rules engine that operates after WAF and bot management policies. Two payment flows are supported: reactive (discovery-first via 402 response) and proactive (intent-first via crawler-max-price header). Billing events are recorded for authenticated requests with payment intent and a crawler-charged header.
More from the Website Security & Threat Management trail
- We tested our own WAF with frontier AI models. Here’s what we found
- When scanners miss the attack: how Cloudflare Client-Side Security protects storefronts
- Introducing automatic remediation policies with Cloudflare CASB
- Introducing context-aware vulnerability discovery and remediation with Cloudflare Managed Defense and OpenAI Daybreak Models
- Introducing Adaptive Intelligence: undermining the economics of every bot attack