CloudFlare and SEO
Jun 26, 2011Matthew Prince
Summary
This post details how Cloudflare addressed an issue where search engine crawlers were reducing crawl rates for sites behind Cloudflare due to issues with a single origin server. Cloudflare developed a new technology to automatically change a site's Cloudflare IP addresses to isolate it from other sites when a problem is detected. They also collaborated with search engine crawl teams to ensure Cloudflare's IP ranges were categorized correctly, allowing search engines to crawl at maximum velocity without overburdening origin servers. This resulted in improved SEO rankings and page speed scores for users.
More from the Website Security & Threat Management trail
- We tested our own WAF with frontier AI models. Here’s what we found
- When scanners miss the attack: how Cloudflare Client-Side Security protects storefronts
- Introducing automatic remediation policies with Cloudflare CASB
- Introducing context-aware vulnerability discovery and remediation with Cloudflare Managed Defense and OpenAI Daybreak Models
- Introducing Adaptive Intelligence: undermining the economics of every bot attack