Reducing flaky builds by 18x

This post details the development of a system to manage flaky tests in CI. The system automatically detects flaky tests by rerunning them three times in different scenarios (same process, shifted into the future, different host) to identify randomness, time-based issues, or order-dependence. It assigns an impact score based on failure frequency and affected branches/developers. High-impact flaky tests trigger automated issue creation, assigned to the most recent author of the test or associated code, with a link to the commit that may have introduced the problem. This system achieved an 18x improvement in reducing commits with flaky builds.