Explore
CloudflareBlogCloudflareBlog CloudflareBlog CloudflareBlog CloudflareBlog CloudflareBlog 
VercelBlogCloudflareBlog CloudflareBlog CloudflareBlog VercelBlog CloudflareBlog
Robots.txt
BotBase for Operators: A clearer path to joining Cloudflare's directory of bots and agents
Say it once: introducing Bot Preference Sync
Introducing the Agent Readiness score. Check to see if your site is agent-ready
Giving users choice with Cloudflare’s new Content Signals Policy
To build a better Internet in the age of AI, we need responsible AI bot principles. Here’s our proposal.
VercelBlogBotID uncovers hidden SEO poisoning
Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives
Control content use for AI training with Cloudflare’s managed robots.txt and blocking for monetized content
No hallucinations here: track the latest AI trends with expanded insights on Cloudflare Radar
VercelBlog
