Test Budget: Time Constrained CI Feedback - Shopify

What this post added

This post details the implementation of a 'Test Budget' system to constrain CI test execution time. It introduces test prioritization techniques (failure_rate, avg_duration, churn, coverage, complexity) to order tests and maximize early failure detection within a fixed time budget. The analysis shows that prioritizing tests based on failure rate allows finding a significant percentage of failures within a reduced test suite execution, enabling faster CI feedback.