How we raised mobile end-to-end test stability to 98% (2026) - Shopify

This post details the development of a new E2E testing framework for Shopify's mobile apps, significantly improving test stability from 50% to 98%. The framework replaces the previous Appium/WebdriverIO setup with an opinionated wrapper that enforces strict testing patterns. Key contributions include: a builder-style API where every step requires an assertion, making flaky tests difficult to write; the use of computer vision (PaddleOCR and OpenCV) for element identification instead of relying solely on Test IDs; and a unified CLI that runs tests consistently across local machines, CI emulators, and remote device farms. The new framework also provides annotated videos for easier debugging and a pre-promotion flakiness gate to ensure new tests are stable before entering the blocking suite.