TechPrufer
Get started
Shopify
Shopify logoShopify · Feature trail

Test Selection and Optimization

8 posts · Apr 2017 – Aug 2026

What this trail covers

Shopify has developed systems to optimize the testing process for large codebases, focusing on improving the stability and reliability of end-to-end (E2E) mobile tests. Initial efforts involved intelligently selecting which tests to run to reduce CI time. This has evolved to building an opinionated wrapper around Appium and WebdriverIO for mobile E2E testing. The new framework enforces strict testing patterns through a builder-style API that requires assertions with every step, preventing flaky tests. It also incorporates computer vision for element identification, moving away from reliance on Test IDs, and provides a unified CLI for running tests across different environments. The goal is to achieve high test stability and enable faster, more confident shipping of mobile features.

All Shopify trails
  1. How we raised mobile end-to-end test stability to 98% (2026) - ShopifyThis post details the development of a new E2E testing framework for Shopify's mobile apps, significantly improving test stability from 50% to 98%. The framework replaces the previous Appium/WebdriverIO setup with an opinionated wrapper that enforces strict testing patterns. Key contributions include: a builder-style API where every step requires an assertion, making flaky tests difficult to write; the use of computer vision (PaddleOCR and OpenCV) for element identification instead of relying solely on Test IDs; and a unified CLI that runs tests consistently across local machines, CI emulators, and remote device farms. The new framework also provides annotated videos for easier debugging and a pre-promotion flakiness gate to ensure new tests are stable before entering the blocking suite.
  2. The Hardest Part of Writing Tests is Getting Started - ShopifyThis post provides a foundational guide for engineers new to testing, detailing the 'why' behind writing tests (documentation, regression prevention, contract enforcement) and offering a four-step approach to getting started: 1. Write a first test to ensure the suite is functional. 2. Test isolated functions first to minimize setup complexity. 3. Use stubs and fixtures to control test environments and avoid testing external dependencies. 4. Set up end-to-end (E2E) tests, particularly for legacy code, to achieve high code coverage and facilitate refactoring.
  3. Test Budget: Time Constrained CI Feedback - ShopifyThis post details the implementation of a 'Test Budget' system for CI feedback. It introduces test prioritization criteria (failure_rate, avg_duration, churn, coverage, complexity, default) and evaluation metrics (TTFF, APFD, Convergence Index) to optimize test execution order within a fixed time constraint. The analysis shows that 'failure_rate' and 'default' prioritization criteria offer the best balance of early failure detection and convergence, enabling the execution of a smaller subset of tests (e.g., 70%) to find a significant percentage of failures.
  4. Keeping Developers Happy with a Fast CI - ShopifyThis post details improvements to the test selection system by adding mappings for non-Ruby files like ActiveRecord fixtures, increasing the percentage of builds that don't select all tests from 45% to over 60%, and improving test stability from 88% to 97%. It also addresses the impact of slow or hanging tests by temporarily disabling them to improve p95 build times.
  5. Spark Joy by Running Fewer Tests - ShopifyThis post details the implementation of a dynamic analysis system for test selection in a large Ruby on Rails monorepo. The system uses Rotoscope and Tracepoint to log method invocations and build a test-to-file mapping. It addresses challenges with dynamically typed languages, metaprogramming, untraceable files, and mapping lag by running dynamic analysis on deployed commits and applying specific criteria for test selection on PRs. It also describes an automatic test disabling mechanism for failing tests and metrics for measuring recall and speed improvement.
  7. iOS Application Testing Strategies at Shopify - ShopifyThis post details Shopify's comprehensive testing strategies for iOS applications within a monorepo architecture. It categorizes tests by component hierarchy (Models, Views, ViewModels, ViewControllers, Flows) and explains the purpose and implementation of each. Techniques discussed include unit testing for models, snapshot testing for views and view controllers to prevent visual regressions, and UI Automation tests (XCUITest) for end-to-end workflow validation. The post also touches upon the importance of accessibility in UI tests and acknowledges the continued need for manual testing for certain types of bugs.
  8. The Unreasonable Effectiveness of Test Retries: An Android Monorepo Case Study - ShopifyThis post details the implementation of test retries to improve CI pass rates in an Android monorepo. It introduces the concept of retriable, fatal, and test failures, with specific retry strategies for each. The post also highlights the importance of caching to reduce reliance on unreliable components and provides recommendations for making components more reliable. The theoretical pass rate of the CI suite increased from 35% to 99.95% by implementing retries.
  9. Building a Dynamic Mobile CI System - ShopifyThis post details the revamp of Shopify's mobile CI system, moving from static configurations to a dynamic, virtualized infrastructure using Buildkite, VMware, and MacStadium's Private Cloud. Key improvements include faster build times (under 10 minutes), enhanced scalability through linked clones and a scheduler/worker VM architecture, and greater flexibility in managing OS and toolchain versions. The system addresses the specific challenges of iOS and Android builds, including code signing and emulator acceleration, and emphasizes the importance of investing in appropriate hardware for efficient CI operations.