Control content use for AI training with Cloudflare’s managed robots.txt and blocking for monetized content
Jul 1, 2025Jin-Hee Lee, Dipunj Gupta, Brian Mitchell, Reid Tatoris, Henry Clausen
Summary
This post introduces two new tools for controlling AI bot access to content: a managed robots.txt file generation service and a feature to block AI bots only on monetized portions of a site. It also provides updated data on AI crawler traffic, crawl-to-refer ratios for major AI companies, and the adoption of AI bot blocking features by Cloudflare customers. The post details the underutilization of robots.txt for AI bots and highlights the shift towards 'fully disallowed' permissions for top AI crawlers.
More from the Website Security & Threat Management trail
- We tested our own WAF with frontier AI models. Here’s what we found
- When scanners miss the attack: how Cloudflare Client-Side Security protects storefronts
- Introducing automatic remediation policies with Cloudflare CASB
- Introducing context-aware vulnerability discovery and remediation with Cloudflare Managed Defense and OpenAI Daybreak Models
- Introducing Adaptive Intelligence: undermining the economics of every bot attack