TechPrufer
Get started
Slack
Slack logoSlack · Feature trail

Continuous Load Testing Infrastructure

4 posts · Jun 2019 – Apr 2022

What this trail covers

This thread tracks the evolution of Slack's load testing capabilities, focusing on enabling continuous load testing to proactively identify performance regressions and build a culture of performance. Initial posts detailed the development of load testing tools like Koi Pond. This post details the implementation of continuous load testing by ensuring clients are always running, integrating load testing into release cycles, and building safety and resilience into the platform. This includes features for load testing Enterprise Key Management (EKM) and the development of a standardized, accessible load testing tool in Go. The tool simplifies provisioning and execution, enabling any engineer to simulate load on APIs and track key metrics like error rates, latency, and database health. The project also emphasizes documentation and onboarding to foster wider adoption and contributions to the tooling.

All Slack trails
  1. Continuous Load TestingThis post details the implementation of continuous load testing by ensuring clients (Koi Pond) are always running in Kubernetes pods. It introduces safety features like Automatic Shutdown (using Trickster and Prometheus metrics to halt tests if thresholds are breached) and an enhanced Emergency Stop service. For resilience, Koi Pond's state is now persisted in DynamoDB, and token generation has been automated using a cron job, a job queue, and encrypted S3 storage with retries. The rollout involved collaboration with infrastructure teams and a gradual ramp-up.
  2. Load Testing with Koi PondThis post details the design and implementation of Koi Pond, a load-testing tool for Slack. It describes how Koi Pond spins up slimmed-down Slack clients ('koi') that establish websocket connections and send API calls, mirroring real client behavior. The post explains two key concepts: 'behaviors' for simulating API usage at realistic frequencies and 'formations' for testing mass user actions like viewing and reacting to messages to simulate 'thundering herd' scenarios. It contrasts Koi Pond with previous tools like Puppet Show and API Blast, highlighting its cost-efficiency and ability to test both backend and real-time services.
  3. Shadow JobsThis post details the design and implementation of 'Shadow Jobs' within GitHub Actions. These jobs run custom CI builds against pre-release versions of core build tools (e.g., Android Gradle Plugin, Kotlin) on a scheduled basis. The post provides a detailed example of a GitHub Actions workflow using a matrix strategy to test various tool versions and configurations, including how to pass these versions as Gradle properties. It also outlines other potential use cases for this pattern, such as testing internal changes or new library dependencies.
  4. 🏄‍♀️ Surf’s Up! Preparing for Huge Waves of Traffic via Load TestingThis post details the strategy and implementation of load testing for Slack's Enterprise Key Management (EKM) service. It outlines 6 steps: identifying critical paths (chat.postMessage), gathering baseline metrics (simulating double peak customer load), creating consistent test environments (tool for spinning up identical orgs), consolidating load testing tools into a single Go-based service with a CLI, automating machine provisioning with a `slack loadtest --bootstrap` command, and tracking metrics (API error rates, EKM latency, database health). It also provides an example of testing the `chat.postMessage` API with the new tool and discusses the importance of onboarding and documentation to drive adoption of the load testing framework across engineering.