Disasterpiece Theater: Slack’s process for approachable Chaos Engineering

This post introduces Slack's 'Disasterpiece Theater' process, a precursor to formal Chaos Engineering. It outlines the methodology for identifying and purposely causing likely failures in production to test fault tolerance without causing user-impacting incidents. The process involves detailed planning, tabletop exercises, hypothesis formulation, phased execution (dev then prod), and debriefing. It highlights three specific exercises: avoiding cache inconsistency in memcached, demonstrating tolerance of zonal failures and network partitions for Channel Servers, and identifying a critical timeout bug in the Confabulator configuration system.