How do you test your tests?

Introduced the Probabilistic Flakiness Score (PFS) to quantify and monitor the reliability of automated regression tests. PFS allows for measuring the degree of flakiness for individual tests and monitoring changes in reliability over time, enabling proactive repair of unreliable tests. The system uses Bayesian inference and a statistical model to estimate flakiness based on observed test results, considering factors like code under test, state of the world, and inherent test flakiness. The model accounts for the asymmetry in how developers perceive passing and failing test results.