More data, more data

This post details the state of processing edge logs (HTTP and DNS) for Cloudflare's Data Team. It highlights the scale of data generated (360TB/day), the infrastructure used (Kafka clusters, CitusDB), and what has worked well (log forwarder, Kafka, Log Share, CitusDB, Platform/SRE support). It also outlines future work, including improving customer-facing service reliability, building new analytics systems (mentioning Spark Streaming), developing new data pipelines, and enhancing tooling for complex analysis.