Cloudflare incident on November 14, 2024, resulting in lost logs

This post details a specific incident where a misconfiguration in the Logfwdr configuration system, combined with a latent 'fail open' bug in Logfwdr and an unprovisioned Buftee system, led to the loss of approximately 55% of customer logs for about 3.5 hours. It describes the architecture of the log delivery pipeline (Logfwdr, Logreceiver, Buftee, Logpush) and the cascading failure modes. The root causes identified are the configuration bug and the failure of Buftee to handle a 40x increase in buffer creation due to unconfigured safeguards. Going forward, Cloudflare is implementing new alerts, fixing the bug and associated tests, and planning regular 'overload tests' to simulate similar cascade failures.