An overview of Cloudflare's logging pipeline

This post details the evolution of Cloudflare's internal logging pipeline, which is a critical component of the Observability Platform. It covers the journey from basic logging to a sophisticated, highly available system capable of shipping close to a million log lines per second. Key contributions include: - Describing the use of various logging libraries (zerolog, KJ_LOG, log) and their integration with systemd-journald. - Explaining the role of syslog-ng for log collection, rate limiting, field rewriting, and JSON wrapping. - Detailing the 'log-x' operation for achieving redundancy across two core data centers (log-a, log-b) and using Kafka Mirror Maker with a Dead Letter Queue (DLQ) to handle transient data center disconnections. - Explaining the use of Kafka for buffering logs, its benefits for consumer flexibility and fault tolerance, and the partitioning strategy using host and service name to guarantee ordering within a partition. - Describing the long-term storage backends: an Elasticsearch/Logstash/Kibana (ELK) stack with dedicated master, data, and HTTP nodes, and a Clickhouse cluster for service logs. - Outlining future projects including multi-tenancy, migration to Open Telemetry Logs, tail sampling, and Kafka cluster balancing.