An overview of Cloudflare's logging pipeline

What this post added

This post details the architecture and operation of Cloudflare's internal logging pipeline, which ships close to a million log lines per second. It covers the journey of logs from service stdout/stderr to systemd-journald, then to syslog-ng for rate limiting and enrichment, and finally to two core data centers (log-a and log-b) for redundancy via Kafka. The post also discusses challenges in log ordering due to syslog limitations, partitioning strategies for Kafka, and long-term storage solutions using Elasticsearch and ClickHouse. Future work includes migrating to Open Telemetry Logs, tail sampling, and improving Kafka cluster balancing.