Over 700 million events/second: How we make sense of too much data

This post details Cloudflare's strategies for handling extremely high-volume data ingestion (706M events/sec) for its Analytics products. It introduces controlled downsampling techniques at various pipeline stages (Logfwdr, Logreceiver, Inserters) to retain information during component failures or overloads. Key technical contributions include: 1. Max-min fairness and weighted stream prioritization in Logfwdr's buffer management using a max-heap and goroutines. 2. Reservoir sampling-like 'bottomless buffers' that thin data automatically. 3. Adaptive sampling in Logreceiver based on customer size, pushing data at multiple resolutions. 4. The application of the Horvitz-Thompson estimator with Poisson sampling for deriving SUM, COUNT, and AVG from downsampled data, including variance estimation and confidence interval calculation. The post also discusses the mathematical underpinnings of these estimators and the challenges of calculating confidence intervals for averages when the total population size is unknown.