Post Mortem: The Ugly, the Bad & the Good

What this post added

This post details a major incident involving a failed DNS infrastructure update that caused significant downtime for customers. It highlights the challenges of DNS propagation, the impact of external caches (like ISPs), and how concurrent DDoS attacks exacerbated the issue. The post also discusses the immediate response, the process of recovery, and the implementation of new safeguards to prevent similar incidents, particularly focusing on improving DNS update speed and hardening against DNS-directed DDoS attacks.