HTTP Analytics for 6M requests per second using ClickHouse

What this post added

This post details the re-architecture of Cloudflare's HTTP traffic analytics pipeline, replacing a legacy PostgreSQL/Citus system with ClickHouse. It describes the challenges of the old pipeline (SPOFs, complexity, high maintenance cost) and the decision to adopt ClickHouse due to its performance, scalability, and fault tolerance. The post covers schema design for ClickHouse, including strategies for handling aggregated data due to high volume (6M requests/sec) and the trade-offs considered for storing raw vs. aggregated data. It also highlights Cloudflare's contributions to ClickHouse, such as the Kafka table engine and aggregate functions.