Flaky Test Detection and Suppression

3 posts · Jan 2018 – Apr 2022

What this trail covers

This thread tracks the evolution of Slack's systems for managing and improving the reliability of automated tests, particularly in mobile codebases. Initial efforts focused on improving test infrastructure and CI stability. This post details the development and implementation of an automated system for detecting and suppressing flaky tests, significantly reducing test job failures and improving main branch stability. Subsequent posts may cover further refinements to the detection algorithms, integration with CI pipelines, and strategies for addressing the root causes of flakiness.

All Slack trails

Apr 2022Handling Flaky Tests at Scale: Auto Detection & SuppressionThis post details the development of an automated system for detecting and suppressing flaky tests in Slack's mobile codebases. It outlines the problem of flaky tests impacting CI stability and developer experience, categorizes flaky tests into independent and systemic issues, and describes the manual triage process. The post then details the first automated approach: a system that parses test results, pulls test history from a backend, calculates flakiness percentage, and suppresses tests exceeding a threshold. It highlights the initial impact on build stability but also discusses drawbacks like leaking failing tests and issues with new tests or failed fixes, leading to a rollback and the need for further iteration.
Feb 2022Balancing Safety and Velocity in CI/CD at SlackThis post details a significant shift in Slack's CI/CD workflow for the Webapp monorepo, moving from predominantly pre-merge testing to a multi-tiered post-merge testing strategy. Key technical changes include the introduction of a post-merge pipeline (executing ~10% of E2E tests) and a regression pipeline (executing remaining E2E tests in batches bi-hourly). This reduced test turnaround time by over 40% and test flakiness per PR by over 90%. The post also describes triage changes, including creating dedicated Slack channels for alerts and assigning test ownership to business areas, and the design of human-centric alerts using a finite state machine to detect consecutive failures.
Jan 2018Android UI Automation: Part 2, Making It EasyThis post introduces the 'PowerPack' pattern to encapsulate the complexity of setting up test data and dependencies for UI tests. It addresses the developer struggle with repetitive mocking by abstracting this setup into model-specific 'Packs' (e.g., UsersPack, ChannelsPack) and an app-wide 'SlackPowerPack'. This pattern simplifies test authoring by allowing developers to focus on test data rather than component mocking. The post also introduces 'PowerPack++', which allows workspace setup via a JSON configuration file, further reducing boilerplate.
Advertisement