Break Stuff on Purpose

This post details a chaos engineering exercise focused on testing and improving Slack's incident response and recovery processes for Kibana and Elasticsearch clusters. The exercise involved intentionally causing a disk space failure on a development Kibana cluster to test new backup and restore procedures, runbooks, and alerting. Key contributions include: identifying and fixing issues with outdated runbooks and backup scripts, improving alert mechanisms, discovering and rectifying firewall rule deficiencies, automating the backup and restore process into a CLI tool, and successfully migrating a Kibana instance to Kubernetes while testing the enhanced recovery process. The exercise highlighted the importance of actively exercising and refining incident response procedures.