Say it once: introducing Bot Preference Sync
Aug 22, 2026Jin-Hee Lee
Summary
Introduces Bot Preference Sync, a feature that automatically updates a site's robots.txt file based on AI bot configuration preferences (Search, Agent, Training) set in the Cloudflare dashboard. This aims to synchronize stated preferences with enforced rules, providing greater transparency and control over how AI crawlers interact with website content, particularly for training and search indexing purposes. It also details new transparency requirements for mixed-use crawlers.
More from the Website Security & Threat Management trail
- We tested our own WAF with frontier AI models. Here’s what we found
- When scanners miss the attack: how Cloudflare Client-Side Security protects storefronts
- Introducing automatic remediation policies with Cloudflare CASB
- Introducing context-aware vulnerability discovery and remediation with Cloudflare Managed Defense and OpenAI Daybreak Models
- Introducing Adaptive Intelligence: undermining the economics of every bot attack