TechPrufer
Get started
Slack
Slack logoSlack · Feature trail

Deploy Safety Program

5 posts · Mar 2020 – Oct 2025

What this trail covers

This thread tracks the evolution of Slack's Deploy Safety Program, aimed at reducing customer impact from changes. Initial posts detail the program's inception, defining the problem of change-triggered incidents, establishing North Star goals for reducing impact time and severity, and introducing the Deploy Safety metric. This post details the foundational architecture for deploying code, including the evolution from simple rsync to a parallel pull-based system and the introduction of atomic deployments and automatic rollbacks. It also covers the investment strategy, focusing on areas of known pain and iterating based on results, and highlights the significant reduction in customer impact hours achieved through these efforts.

All Slack trails
  1. Deploy Safety: Reducing customer impact from changeThis post details the evolution of Slack's Deploy Safety Program, focusing on the investment strategy and the impact of automatic rollbacks. It highlights the shift from manual processes to automated detection and remediation, the development of a Deploy Safety metric (Hours of customer impact from high severity and selected medium severity change-triggered incidents), and the iterative approach to project investment. The post emphasizes the importance of patience with trailing metrics and the critical role of direct outreach to engineering teams for adoption. It also showcases the dramatic improvement in results after automatic rollbacks were introduced, leading to a significant reduction in customer impact hours.
  2. Break Stuff on PurposeThis post details a chaos engineering exercise focused on testing and improving Slack's incident response and recovery processes for Kibana and Elasticsearch clusters. The exercise involved intentionally causing a disk space failure on a development Kibana cluster to test new backup and restore procedures, runbooks, and alerting. Key contributions include: identifying and fixing issues with outdated runbooks and backup scripts, improving alert mechanisms, discovering and rectifying firewall rule deficiencies, automating the backup and restore process into a CLI tool, and successfully migrating a Kibana instance to Kubernetes while testing the enhanced recovery process. The exercise highlighted the importance of actively exercising and refining incident response procedures.
  3. The Scary Thing About Automating DeploysThis post details the implementation of an automated deployment system, ReleaseBot, which replaces manual Deploy Commanders. The core of ReleaseBot's safety mechanism is anomaly detection, using z-scores and dynamic thresholds to monitor system metrics during deployments. This approach allows for faster and more vigilant detection of issues compared to human monitoring, thereby increasing deployment safety and freeing up valuable engineering time. The post explains the mathematical basis of z-scores and provides examples of their implementation in Python and PromQL.
  4. Slowing Down to Speed Up – Circuit Breakers for Slack’s CI/CDThis post details the implementation of orchestration-level circuit breakers within Slack's CI/CD service (Checkpoint) to address cascading failures caused by internal request stampedes. It describes the problem of internal services being stretched to their limits due to growth, leading to degraded developer workflows. The solution involved applying circuit breaker patterns to defer or shed requests in Checkpoint, using Trickster for Prometheus metric lookups, and managing circuit breaker states to resume work upon service recovery. This approach reduced cascading failure incidents and flaky CI executions.
  5. Deploys at SlackThis post details the evolution of Slack's deployment system, moving from a simple rsync-based model to a parallel pull-based system for faster deploys and introducing atomic deploys using hot/cold directories to prevent errors during file copying. It also outlines the current tiered rollout process (staging, dogfood, canary, percentage-based) managed by deploy commanders, emphasizing the shift towards reliability and proactive error detection.