How Cloudflare analyzes 1M DNS queries per second

What this post added

This post details the implementation of a real-time analytics system for Cloudflare DNS logs, focusing on the challenges of handling high-volume, high-cardinality data. It describes the data pipeline from edge log forwarding via Cap'n Proto and Kafka to storage and analysis using ClickHouse. Key technical contributions include the development of a Go adapter for Kafka-to-ClickHouse ingestion, optimization of ClickHouse ingestion performance through batching, and careful selection of primary keys and materialized views for efficient querying of zone-level data and aggregations. The post also discusses the limitations of aggregated data and the necessity of unaggregated logs for root cause analysis.