More details about the October 4 outage

This post details a major outage caused by a command issued during routine maintenance of Meta's global backbone network capacity. The command unintentionally took down all backbone connections, effectively disconnecting Facebook data centers globally due to a bug in the audit tool. This led to a secondary issue where DNS servers withdrew BGP advertisements, making them unreachable. The recovery process was complicated by the inaccessibility of data centers and the failure of internal DNS-dependent tools. Onsite engineers were dispatched to debug and restart systems, a process that took time due to physical and system security measures. The company leveraged "storm" drills to manage the load surge when bringing services back online. The post emphasizes learning from the incident, strengthening testing and drills, and acknowledges the trade-off between security hardening and recovery speed.