Lessons Learned from Scaling Up Cloudflare’s Anomaly Detection Platform
Mar 12, 2021Jeffrey Tang
Summary
This post details the development and scaling of Cloudflare's Anomaly Detection platform, a key component of its Bot Management system. It explains the underlying technology (HBOS, ClickHouse, Redis, HyperLogLog), the platform's architecture (microservices), and the lessons learned during its evolution from a monolithic service to a highly scalable system capable of processing 500K requests per second. Key contributions include insights into optimizing Redis performance (PFMERGE, recency register, key encoding) and the strategic shift to a microservices architecture to improve resilience and reduce load on shared dependencies.
More from the Website Security & Threat Management trail
- We tested our own WAF with frontier AI models. Here’s what we found
- When scanners miss the attack: how Cloudflare Client-Side Security protects storefronts
- Introducing automatic remediation policies with Cloudflare CASB
- Introducing context-aware vulnerability discovery and remediation with Cloudflare Managed Defense and OpenAI Daybreak Models
- Introducing Adaptive Intelligence: undermining the economics of every bot attack