TechPrufer
Get started
GitHub
GitHub logoGitHub · Feature trail

Flaky Test Management System

1 post · Dec 2020

What this trail covers

This feature thread tracks the evolution of managing flaky tests in CI systems. Initially, flaky tests caused significant delays and frustration, with a high percentage of commits affected. This post details the development and implementation of an automated system to detect, contain, and prioritize the investigation of flaky tests. The system identifies flaky tests by rerunning them in different scenarios (same process, shifted into the future, different host) and assigns issues to the most likely author based on failure history and git blame. This has led to a dramatic reduction in the impact of flaky tests on deployment velocity.

All GitHub trails
  1. Reducing flaky builds by 18xThis post details the development of a system to manage flaky tests in CI. The system automatically detects flaky tests by rerunning them three times in different scenarios (same process, shifted into the future, different host) to identify randomness, time-based issues, or order-dependence. It assigns an impact score based on failure frequency and affected branches/developers. High-impact flaky tests trigger automated issue creation, assigned to the most recent author of the test or associated code, with a link to the commit that may have introduced the problem. This system achieved an 18x improvement in reducing commits with flaky builds.