Test Budget: Time Constrained CI Feedback - Shopify

This post details the implementation of a 'Test Budget' system for CI feedback. It introduces test prioritization criteria (failure_rate, avg_duration, churn, coverage, complexity, default) and evaluation metrics (TTFF, APFD, Convergence Index) to optimize test execution order within a fixed time constraint. The analysis shows that 'failure_rate' and 'default' prioritization criteria offer the best balance of early failure detection and convergence, enabling the execution of a smaller subset of tests (e.g., 70%) to find a significant percentage of failures.