HTTP Analytics for 6M requests per second using ClickHouse

This post details the migration of Cloudflare's HTTP traffic analytics pipeline from a legacy PostgreSQL/Citus-based system to ClickHouse. The new pipeline addresses limitations of the old system, including single points of failure, complex codebase, and high maintenance costs. It leverages ClickHouse for its columnar storage, real-time analytical capabilities, horizontal scalability, and fault tolerance. The post discusses schema design for aggregated data, performance tuning of ClickHouse, and the integration of the Kafka table engine for direct ingestion.