Flaky Test Management in Azure Pipelines

7 posts · Oct 2015 – May 2023

What this trail covers

Azure Pipelines Test Analytics provides insights into test failures, pass rates, and trends. It helps identify flaky tests, problematic test files, and owners, and allows for grouping and filtering data by various pivots like stage, branch, and owner. It also supports measuring test quality upstream and optimizing test efficiency by identifying long-running tests. The feature aims to improve pipeline health and shorten release cycles by addressing test debt and unreliable tests. This post introd

All Microsoft trails

May 2023Improved Flaky Test Management in Azure PipelinesIntroduced the ability to delete past history of flaky test cases when changing the Flaky Test management settings. A confirmation alert is added to inform users of the impact of the setting change.
Oct 2018Test Analytics in Azure Pipelines is now at your fingertipsThis post introduces and details the Test Analytics feature in Azure Pipelines, which provides near real-time visibility into continuous testing for build and release pipelines. It highlights the 'Test failure report' and 'Specific test report' with key insights such as pass percentage, unique failing tests, failure trends, and debug information. The post also outlines several scenarios where this feature can be used, including assessing continuous testing health, finding trends to identify what to fix, uncovering hidden patterns impacting team productivity, measuring test quality upstream, filtering reports for specific areas, and making tests more efficient by analyzing average duration.
Oct 2018Test Analytics in Azure Pipelines is now at your fingertipsThis post introduces the Test Analytics feature in Azure Pipelines, providing near real-time visibility into continuous testing for build and release pipelines. It offers a 'Test failure report' with insights into pass percentage, unique failing tests, failure trends, and the ability to group and filter data by pivots like test file, owner, and stage. A 'Specific test report' provides trend charts, execution instances, debug information, and filtering by branch or owner. The post highlights scenarios for measuring continuous testing health, identifying trends for fixing issues, uncovering patterns impacting team productivity, measuring test quality upstream ('shift left'), viewing test reports for specific areas, and making tests more efficient by analyzing average duration.
Nov 2017Test batching options with VSTest taskThis post introduces and details three options for test batching within the VSTest task in Azure Pipelines: 1. Based on the number of tests and agents, allowing for equal test counts per batch or a specified batch size. 2. Based on the past running time of tests, grouping tests into batches of approximately equal execution times derived from previous runs, with specific considerations for unknown test execution times. 3. Based on test assemblies, grouping tests from the same assembly into a single batch. It also notes feature availability across VSTS, TFS 2018+, and upcoming updates, and the distinction between build and release pipeline capabilities.
Oct 2016Tracking branch health and identifying flaky tests in RM driven test automationThis post details the use of Release Management's branch filters to track the health of specific branches and the 'View History' feature to pinpoint the exact commit that caused a test to start failing. It also outlines how to differentiate between test failures caused by developer check-ins, infrastructure issues, and flaky tests.
Advertisement
Oct 2015Test result data retention with Team Foundation Server 2015This post introduces test result data cleanup capabilities in TFS 2015 Update 1. It enables deleting test results when builds are deleted (for Build vNext) and cleaning up test results based on age at the team project level. This includes team project-level retention policies for automated and manual test results, and build vNext-level policies to delete test results when builds are deleted. These features aim to keep test systems performant and responsive by periodically cleaning up irrelevant test data.