Test Selection and Optimization

2/2/2023

This post provides a foundational guide for engineers new to writing tests, addressing the common challenge of getting started. It outlines the 'why' behind testing (documentation, regression prevention, contract enforcement) and offers a four-step approach: writing the first test, focusing on isolated functions, utilizing stubs and fixtures for controlled environments, and implementing end-to-end tests. It emphasizes the importance of ensuring tests can fail and the value of testing legacy code through E2E tests before refactoring.

3/7/2022

This post details the implementation of a 'Test Budget' system to constrain CI test execution time. It introduces test prioritization techniques (failure_rate, avg_duration, churn, coverage, complexity) to order tests and maximize early failure detection within a fixed time budget. The analysis shows that prioritizing tests based on failure rate allows finding a significant percentage of failures within a reduced test suite execution, enabling faster CI feedback.

2/24/2021

This post details improvements to Shopify's CI test selection process. It expands the existing test selection system to include mappings for non-Ruby files, specifically ActiveRecord fixtures, by subscribing to ActiveSupport notifications for SQL queries. This increases the percentage of builds that don't run the full test suite from 45% to over 60% and improves test stability from 88% to 97%. The post also discusses strategies for handling slow or problematic tests, including temporary disabling, which reduced the p95 CI build time by 10 minutes. Additionally, it highlights improvements in agent preparation by addressing Docker start time bottlenecks related to I/O, reducing p95 Docker start time from 90 seconds to 25 seconds, and optimizing dependency building by avoiding unnecessary runs through hash-based caching for database migrations and asset compilation, reducing that job's time from 5 minutes to 3 minutes.

6/11/2020

This post details the implementation of a dynamic analysis system to select relevant tests for execution based on code changes. It describes the challenges of a large, dynamically typed Ruby codebase, the use of Rotoscope and Tracepoint for call graph logging, and the trade-offs of dynamic analysis. The post also covers the rollout strategy, metrics for success (failure recall, speed improvement, compute time savings), and other explored approaches like static analysis and machine learning. It highlights the benefits of reduced test execution time, fewer intermittently failing tests, and lower CI costs.

1/15/2019

This post details Shopify's comprehensive testing strategies for iOS applications within a monorepo architecture. It categorizes tests by component hierarchy (Models, Views, ViewModels, ViewControllers, Flows) and explains the purpose and implementation of each. Techniques discussed include unit testing for models, snapshot testing for views and view controllers to prevent visual regressions, and UI Automation tests (XCUITest) for end-to-end workflow validation. The post also touches upon the importance of accessibility in UI tests and acknowledges the continued need for manual testing for certain types of bugs.

1/8/2019

This post details the implementation of test retries as a strategy to combat flakiness in the Android monorepo's CI system. It quantifies the impact of flakiness on CI pass rates and demonstrates how retries, applied to retriable failures, fatal failures, and test failures (up to three times for individual tests), can significantly increase the theoretical pass rate from 35% to 99.95%. The post also highlights the importance of caching for Gradle builds and provides tips for reducing flakiness, such as avoiding unreliable components and making them more reliable. The successful application of these strategies led to an increase in the Android app pipeline pass rate from 31% to almost 90%.

4/13/2017

This post details the revamp of Shopify's mobile CI system, moving from static configurations to a dynamic, virtualized infrastructure using Buildkite, VMware, and MacStadium's Private Cloud. Key improvements include faster build times (under 10 minutes), enhanced scalability through linked clones and a scheduler/worker VM architecture, and greater flexibility in managing OS and toolchain versions. The system addresses the specific challenges of iOS and Android builds, including code signing and emulator acceleration, and emphasizes the importance of investing in appropriate hardware for efficient CI operations.