Handling Flaky Tests at Scale: Auto Detection & Suppression

This post details the development of an automated system for detecting and suppressing flaky tests in Slack's mobile codebases. It outlines the problem of flaky tests impacting CI stability and developer experience, categorizes flaky tests into independent and systemic issues, and describes the manual triage process. The post then details the first automated approach: a system that parses test results, pulls test history from a backend, calculates flakiness percentage, and suppresses tests exceeding a threshold. It highlights the initial impact on build stability but also discusses drawbacks like leaking failing tests and issues with new tests or failed fixes, leading to a rollback and the need for further iteration.