Taming Test Flakiness: How We Built a Scalable Tool to Detect and Manage Flaky Tests

This post introduces Flakinator, a new platformized, tech-stack agnostic tool developed to detect, manage, and mitigate flaky tests across Atlassian's codebases. It details the hidden costs of flaky tests, citing Atlassian's internal data (21% of master build failures in Jira Frontend, 15% of Jira backend repo failures leading to 150,000+ wasted developer hours annually). Flakinator's capabilities include efficient identification using advanced algorithms and ML, quarantine mechanisms, trend analysis and reporting via dashboards, root cause analysis, custom settings, scalability, and integration with Jira and Slack. The design overview shows Flakinator ingesting test run data from CI systems, storing raw test data, and employing various detection mechanisms. The ecosystem works by identifying flaky tests, assigning owners, creating Jira tickets, and sending Slack notifications. It handles over 350 million test executions daily and stores over 3TB of data. Detection algorithms include a retry mechanism (81% detection rate for some products) and Bayesian inference for calculating a flakiness score (0-1) based on historical analysis, signal processors, and scoring. The post also details the architecture with an ingestion pipeline, flakiness detection engine, and notification/insights module.