A Terrible, Horrible, No-Good, Very Bad Day at Slack

This post details the root cause of the May 12th, 2020 outage, which was a bug in the `haproxy-server-state-management` program that led to stale HAProxy backend configurations. The bug occurred because the program attempted to find slots for new webapp instances before freeing slots for old ones, causing it to fail and exit early when no slots were available. This resulted in HAProxy instances not receiving updated backend lists, especially during traffic scaling events. The outage was mitigated by a rolling restart of the HAProxy fleet. The post also highlights the ongoing migration from HAProxy to Envoy Proxy for ingress load balancing, noting that Envoy's native integration with an xDS control plane will prevent similar issues.