Site Reliability and Outage Management
How do you test your tests?

How do you test your tests?

12/10/2020 · Mateusz Machalica, Wojtek Chmiel, Stanislaw Swierc, Ruslan Sakevych

What this post added

Introduced the Probabilistic Flakiness Score (PFS) to quantify and monitor the reliability of automated regression tests. PFS allows for measuring the degree of flakiness for individual tests and monitoring changes in reliability over time, enabling proactive repair of unreliable tests. The system uses Bayesian inference and a statistical model to estimate flakiness based on observed test results, considering factors like code under test, state of the world, and inherent test flakiness. The model accounts for the asymmetry in how developers perceive passing and failing test results.

Read the original post ↗