TechPrufer
Get started
Atlassian
Atlassian logoAtlassian · Feature trail

Bitbucket Test Health Dashboards

7 posts · Jul 2022 – Sep 2026

What this trail covers

Bitbucket Tests provides visibility into test reliability, historical trends, and suite breakdowns, including flaky and quarantined tests. New dashboards offer an overview of test health, with drill-down capabilities into individual test execution history and summary. The system integrates with Bitbucket Pipelines by consuming JUnit-compatible test reports. This post extends the capability to automatically detect and quarantine flaky tests, reducing CI noise and improving build reliability by enabling the development of a scalable tool called Flakinator to detect, manage, and mitigate flaky tests across all codebases effectively. Flakinator utilizes advanced algorithms and machine learning for efficient identification, quarantine mechanisms, trend analysis, root cause analysis, custom settings, scalability, and seamless integration with other tools like Jira and Slack. It handles over 350 million test executions per day and stores over 3TB of data. Detection algorithms include a retry mechanism with an 81% detection rate for certain products and Bayesian inference for calculating a flakiness score.

All Atlassian trails
  1. Monitor test health at a glance in Bitbucket TestsIntroduces new test health dashboards in Bitbucket Tests, providing visibility into test reliability, historical trends, high-level suite breakdowns (total, healthy, flaky, and quarantined tests), and a dedicated 'Needs attention' card. Enhances integration with Bitbucket Pipelines by linking results to Bitbucket Tests for detailed history and execution analysis. Provides filters and navigation for focused investigation of test results.
  2. Bitbucket Tests now supports tracking tests from multiple branchesThis post extends the Bitbucket Tests capability by enabling tracking of tests from multiple branches (up to 5 per repository, including the default branch). Previously, only the default branch was tracked. This change allows failure rates and flaky scores to aggregate across all tracked branches, providing a more accurate reflection of real delivery workflows and enabling branch-specific filtering of test results.
  3. Fix flaky tests with AI, and track future test work in JiraThis post introduces three new capabilities to the Bitbucket Tests feature: AI-powered flaky test fixing via a draft pull request, direct creation of Jira work items from test insights with pre-filled details, and the ability to remove stale test entries without deleting the test from the repository. The AI fix leverages a 'fix-flaky-test-agent' with a custom prompt and a 'fix-flaky-test-pipeline'. The Jira integration opens a modal within Bitbucket to create issues, linking them back to the test. Removing tests declutters metrics without data loss.
  4. Introducing Flaky Test Detection in Bitbucket TestsIntroduces automatic detection and quarantine of flaky tests within Bitbucket Pipelines. This feature analyzes test history to flag unstable tests and optionally auto-quarantines them to prevent blocking builds. It builds upon the existing Bitbucket Tests framework by adding an intelligent layer for identifying and managing test flakiness without manual intervention.
  5. [Open beta] Introducing Tests in Bitbucket PipelinesThis post introduces the open beta of a new 'Tests' feature in Bitbucket Pipelines. It allows users to organize, track, and optimize tests directly within their repository. Key features include test summaries with failure rate, average duration, and variance; detailed test execution history; and the ability to mark tests as 'Flaky' or 'Quarantined'. The system supports skipping quarantined tests in pipelines by downloading a test metadata file. It consumes JUnit-style XML reports from various frameworks.
  7. Taming Test Flakiness: How We Built a Scalable Tool to Detect and Manage Flaky TestsThis post introduces Flakinator, a new platformized, tech-stack agnostic tool developed to detect, manage, and mitigate flaky tests across Atlassian's codebases. It details the hidden costs of flaky tests, citing Atlassian's internal data (21% of master build failures in Jira Frontend, 15% of Jira backend repo failures leading to 150,000+ wasted developer hours annually). Flakinator's capabilities include efficient identification using advanced algorithms and ML, quarantine mechanisms, trend analysis and reporting via dashboards, root cause analysis, custom settings, scalability, and integration with Jira and Slack. The design overview shows Flakinator ingesting test run data from CI systems, storing raw test data, and employing various detection mechanisms. The ecosystem works by identifying flaky tests, assigning owners, creating Jira tickets, and sending Slack notifications. It handles over 350 million test executions daily and stores over 3TB of data. Detection algorithms include a retry mechanism (81% detection rate for some products) and Bayesian inference for calculating a flakiness score (0-1) based on historical analysis, signal processors, and scoring. The post also details the architecture with an ingestion pipeline, flakiness detection engine, and notification/insights module.
  8. More reliable merge checksThis post introduces a change to Bitbucket's pull request merge checks to prevent merging while builds are in progress. Previously, pull requests could be merged even if builds were still running and subsequently failed. The enhancement adds a validation step to ensure all builds have completed successfully before allowing a merge, thereby increasing the reliability of build merge checks.