Log analytics using ClickHouse

This post details the migration of Cloudflare's error logging pipeline from Elasticsearch to ClickHouse. It outlines the challenges faced with Elasticsearch, including mapping explosion, poor multi-tenancy support, operational overhead, and garbage collection issues, especially at Cloudflare's scale of 35-45 million HTTP requests per second with 500K-800K errors per second. The solution involves adopting ClickHouse, a column-oriented database, to improve query performance, reduce storage costs, and enable the storage of all error logs without sampling. Key ClickHouse features leveraged include its column-oriented storage for fast sequential scans, design for analytical workloads with many columns, sparse indexing, efficient LZ4 compression with configurable codecs (e.g., Double-Delta for DateTime, Gorilla for Float, LowCardinality for String), and linear scalability. An efficient inserter using Cap'n Proto messages and optimized batch sizes is also described.