The Internet has a second audience

This post details the significant shift in internet traffic composition, with AI agents now constituting over half of HTTP requests. It introduces Cloudflare's strategies and products to address this change: AI Crawl Control, Business Insights, BotBase, and Web Bot Auth for identifying and verifying AI agents; granular controls for search, agent, and training traffic; and the 'Disallow AI Training' mechanism. Crucially, it introduces two new monetization primitives: Pay Per Use, for direct publisher-AI deals and content licensing, and Monetization Gateway, for per-request pricing of APIs and services. These aim to create a 'payable' internet where content creators can monetize agent usage, moving beyond a binary 'yes' or 'no' to content access.
More from the Rate Limiting and Event Counting trail
- Follow the thread: a new dashboard to investigate account abuse
- Saving another 100TB of RAM with math (and Rust)
- Have it both ways: stay discoverable in search while disallowing AI training
- Introducing Adaptive Intelligence: undermining the economics of every bot attack
- BotBase for Operators: A clearer path to joining Cloudflare's directory of bots and agents