Control content use for AI training with Cloudflare’s managed robots.txt and blocking for monetized content

What this post added

Introduces two new tools for website owners to control AI bot access to their content: a managed robots.txt file service and an option to block AI bots only on monetized portions of a site. This addresses the shift in the crawler relationship from symbiotic to extractive, evidenced by high crawl-to-referral ratios for AI bots. The post also provides updated data on AI bot traffic and usage of Cloudflare's existing AI bot blocking features, highlighting the underutilization of robots.txt for AI crawlers and the increasing adoption of Cloudflare's AI bot management tools.