Spark Joy by Running Fewer Tests - Shopify

What this post added

This post details the implementation of a dynamic analysis system to select relevant tests for execution based on code changes. It describes the challenges of a large, dynamically typed Ruby codebase, the use of Rotoscope and Tracepoint for call graph logging, and the trade-offs of dynamic analysis. The post also covers the rollout strategy, metrics for success (failure recall, speed improvement, compute time savings), and other explored approaches like static analysis and machine learning. It highlights the benefits of reduced test execution time, fewer intermittently failing tests, and lower CI costs.