Over 700 million events/second: How we make sense of too much data

What this post added

This post details Cloudflare's techniques for handling over 700 million events per second for its analytics products. It introduces 'bottomless buffers' for controlled data downsampling using max-heaps and max-min fairness to manage buffer overflow. It also explains adaptive sampling in Logreceiver to improve accuracy for small customers and aggressively downsample large ones. The post elaborates on using the Horvitz-Thompson estimator with Poisson sampling to derive analytics (SUM, COUNT, AVG) from downsampled data and calculate confidence intervals, addressing challenges like unknown population size.