TechPrufer
Get started
Spotify
Spotify logoSpotify · Feature trail

Test Flakiness Management

1 post · Oct 2019

What this trail covers

Spotify has developed and continues to evolve systems for identifying, tracking, and mitigating test flakiness. This includes visualization tools like Odeneye to distinguish between test flakiness and infrastructure issues, simple reporting tables that have shown significant reductions in flakiness, and automated bots like Flakybot to provide pre-merge confidence in test stability. The ongoing effort aims to improve overall test reliability and engineer confidence in continuous delivery.

All Spotify trails
  1. Test Flakiness – Methods for identifying and dealing with flaky testsThis post details Spotify's journey in addressing test flakiness, introducing three key systems: Odeneye for visualizing test suite results and differentiating flakiness from infrastructure problems, a simple reporting table that demonstrably reduced flakiness by making results visible, and Flakybot for pre-merge validation of test stability. It also outlines common causes of flakiness such as inconsistent assertion timing, reliance on test order, and the inherent brittleness of end-to-end tests.