Site Reliability and Outage Management
More details about the October 4 outage

More details about the October 4 outage

10/5/2021 · Santosh Janardhan

What this post added

This post details the technical causes and recovery process of the October 4th outage, which was triggered by a command that unintentionally took down the global backbone network. It explains the role of routers, DNS, and BGP in network connectivity, the challenges faced due to the loss of both primary and out-of-band network access, and the physical security measures that impacted recovery time. The post also discusses the 'storm' drills used to manage service restoration and the lessons learned regarding the balance between security hardening and recovery speed.

Read the original post ↗