More data, more data

What this post added

This post details the scale of edge log processing at Cloudflare, handling millions of HTTP and DNS logs per second, resulting in hundreds of terabytes of raw data daily. It highlights the infrastructure used, including Kafka clusters and CitusDB, and discusses what has worked well (log forwarder, Kafka, Log Share persistence/retrieval, CitusDB, platform/SRE support) and what remains to be done (improving service reliability for customers, new analytics systems, new data pipelines, better support for complex analysis).