
6/26/2024 · Lloyd Wallis, Julien Desgats, Manish Arora
What this post added
This post details a specific incident on June 20, 2024, caused by two independent events: suboptimal network traffic re-routing due to automated monitoring and a latent bug in the rate limiting system triggered by a new DDoS mitigation mechanism. The latter event, where a specific HTTP request caused a process to enter an infinite loop, is described in technical detail, including the root cause involving a recursive function call due to an anticipated internal API usage and a broken cookie validation function. The post also outlines the incident timeline, impact metrics (error rates, TTFB latency), and the mitigation steps taken, such as disabling the DDoS rule and restarting services.