Scaling Security Insights: how we achieved a 10x increase in global scanning capacity

This post details how Cloudflare achieved a 10x increase in global scanning capacity for its Security Insights feature. It describes the technical challenges and solutions implemented, including scaling Kafka by introducing batch processing and parallel goroutines, avoiding head-of-line blocking with a 'slow lane' and 'fast lane' approach, optimizing database inserts using a hybrid UNNEST/COPY strategy, resolving API latency issues by switching to an active-passive configuration, and rethinking the scheduler with independent zone scheduling, randomized timing, and adaptive rate limiting to ensure uniform scan distribution and handle increased scanning frequency for all customers.