How Cloudflare analyzes 1M DNS queries per second

This post details the technical evolution of Cloudflare's DNS analytics pipeline. It describes the use of Cap'n Proto for edge log generation, Apache Kafka for buffering, and the adoption of ClickHouse as an OLAP system for storing and querying DNS telemetry data. Key technical details include the ingestion adapter service written in Go, ClickHouse's MergeTree storage engine, columnar storage, sparse indexing, and the careful selection of primary keys for optimized read performance. The post also discusses the challenges of aggregating high-cardinality data and the decision to use materialized views for predefined aggregations.