The Unreasonable Effectiveness of Test Retries: An Android Monorepo Case Study - Shopify

What this post added

This post details the implementation of test retries as a strategy to combat flakiness in the Android monorepo's CI system. It quantifies the impact of flakiness on CI pass rates and demonstrates how retries, applied to retriable failures, fatal failures, and test failures (up to three times for individual tests), can significantly increase the theoretical pass rate from 35% to 99.95%. The post also highlights the importance of caching for Gradle builds and provides tips for reducing flakiness, such as avoiding unreliable components and making them more reliable. The successful application of these strategies led to an increase in the Android app pipeline pass rate from 31% to almost 90%.