Performance Testing At Scale—for BFCM and Beyond - Shopify

This post details Shopify's approach to BFCM Scale Testing, a critical exercise for preparing the e-commerce platform for peak traffic. It outlines the process of scaling up core platform components and select subsystems, running various load tests (browsing/buying, admin, flash sale, storefront API, Hydrogen/Oxygen) using the Cronograma tool, and analyzing results. Key challenges and solutions are discussed, including managing cloud vendor resource availability for large-scale upscaling, addressing connection-count-based issues by scaling dependent services or refactoring, and determining test targets based on projections and actuals. The post also covers the importance of scaling down in reverse order of scale-up and conducting blameless retrospectives. It contrasts this full-scale testing approach with the scaled-model strategy, highlighting the former's benefits in identifying issues at scale and the latter's drawbacks.