Slowing Down to Speed Up – Circuit Breakers for Slack’s CI/CD

This post details the implementation of orchestration-level circuit breakers within Slack's CI/CD service (Checkpoint) to address cascading failures caused by internal request stampedes. It describes the problem of internal services being stretched to their limits due to growth, leading to degraded developer workflows. The solution involved applying circuit breaker patterns to defer or shed requests in Checkpoint, using Trickster for Prometheus metric lookups, and managing circuit breaker states to resume work upon service recovery. This approach reduced cascading failure incidents and flaky CI executions.