Log analytics using ClickHouse

What this post added

This post details the migration of Cloudflare's error logging pipeline from Elasticsearch to ClickHouse. It outlines the challenges faced with Elasticsearch, including mapping explosion, poor multi-tenancy support, operational overhead, and garbage collection issues, especially at Cloudflare's scale of 35-45 million HTTP requests per second with 500K-800K errors per second. The solution involves adopting ClickHouse, a column-oriented database, to improve query performance, reduce storage costs, and enable the storage of all error logs without sampling. Key ClickHouse features leveraged include its column-oriented storage for fast sequential scans, design for analytical workloads with many columns, sparse indexing, efficient LZ4 compression with configurable codecs (e.g., Double-Delta for DateTime, Gorilla for Float, LowCardinality for String), and linear scalability. An efficient inserter using Cap'n Proto messages and optimized batch sizes is also described.