TechPrufer
Get started
Slack
Slack logoSlack · Feature trail

Chaos Engineering Program

1 post · Aug 2019 – Dec 2024

What this trail covers

This thread tracks the evolution of Slack's Chaos Engineering program, aimed at proactively testing and improving the resilience and fault tolerance of its systems. Initial posts detail the inception of the 'Disasterpiece Theater' process, focusing on safe, regular, and approachable testing of critical systems in production. This includes detailed planning, tabletop exercises, and phased rollouts from development to production environments. This post details a specific exercise where a development Kibana cluster was intentionally failed to test backup and restore processes, leading to improvements in runbooks, alerting, and automation of the backup/restore tooling. A subsequent test involved migrating a Kibana instance to Kubernetes and testing the improved backup script.

All Slack trails
  1. Break Stuff on PurposeThis post details a specific chaos engineering exercise where a development Kibana cluster was intentionally failed by filling its disk to test backup and restore processes. This exercise identified issues with outdated runbooks, lack of backup alerts, and failed restore attempts, leading to improvements in scheduled backup scripts, runbook accuracy, and S3 retention policies. The exercise also uncovered firewall rule needs. A subsequent test involved migrating a Kibana instance to Kubernetes and successfully testing the improved backup script with ~30 minutes of downtime. The team also automated the backup/restore process into a full-featured CLI program.
  2. Disasterpiece Theater: Slack’s process for approachable Chaos EngineeringThis post introduces Slack's 'Disasterpiece Theater' process, a precursor to formal Chaos Engineering. It outlines the methodology for identifying and purposely causing likely failures in production to test fault tolerance without causing user-impacting incidents. The process involves detailed planning, tabletop exercises, hypothesis formulation, phased execution (dev then prod), and debriefing. It highlights three specific exercises: avoiding cache inconsistency in memcached, demonstrating tolerance of zonal failures and network partitions for Channel Servers, and identifying a critical timeout bug in the Confabulator configuration system.