TechPrufer
Get started
Shopify
Shopify logoShopify · Feature trail

Resiliency Planning and Game Days

15 posts · Jan 2015 – Nov 2025

What this trail covers

Shopify has evolved its resiliency planning and operational readiness, particularly for high-traffic events like Black Friday Cyber Monday (BFCM). Initial efforts focused on load and stress testing to identify system limits and regressions. This has evolved to include capacity planning based on traffic forecasts and historical data, with detailed resource estimations for key areas and buffers for flexibility. Scalability testing, including "faux" BFCM events like "Oktoberfest scale-up", is now a core part of the readiness program. This includes extensive Game Days simulating production failures at BFCM scale, cross-system disaster simulations, and load testing on production infrastructure across multiple regions. New challenges arise from preparing infrastructure that has not experienced holiday traffic, requiring systematic mapping of system behavior under stress and optimization of ETL pipelines and API layers. Scale testing validates the entire platform working together at BFCM volumes, revealing issues that only surface when everything runs at capacity simultaneously. The operational plan coordinates engineering teams, incident response, and live system tuning with real-time monitoring, incident response teams, merchant communications, and live optimization. Tools and processes developed for BFCM readiness, such as the Resiliency Matrix and Critical Journey Game Days, are permanent infrastructure improvements that enhance daily resilience.

All Shopify trails
  1. How we prepare Shopify for BFCM (2025) - ShopifyThis post details Shopify's comprehensive BFCM (Black Friday Cyber Monday) readiness program for 2025, focusing on year-round resilience building, Game Days (chaos engineering at scale), load testing, and scale testing. It highlights the challenges of preparing new infrastructure for peak traffic, the systematic approach to identifying and fixing bottlenecks (e.g., Kafka partition increases, API memory optimization, connection timeout tuning), and the integration of these efforts into a permanent operational plan. The post emphasizes the scale of testing, including simulating regional failovers and coordinating with external providers like YouTube, and the iterative process of fixing issues identified across multiple scale tests. It also touches on the importance of cross-team coordination and the development of permanent infrastructure improvements like the Resiliency Matrix and Critical Journey Game Days.
  2. Performance Testing At Scale—for BFCM and Beyond - ShopifyThis post details Shopify's approach to BFCM Scale Testing, a critical exercise for preparing the e-commerce platform for peak traffic. It outlines the process of scaling up core platform components and select subsystems, running various load tests (browsing/buying, admin, flash sale, storefront API, Hydrogen/Oxygen) using the Cronograma tool, and analyzing results. Key challenges and solutions are discussed, including managing cloud vendor resource availability for large-scale upscaling, addressing connection-count-based issues by scaling dependent services or refactoring, and determining test targets based on projections and actuals. The post also covers the importance of scaling down in reverse order of scale-up and conducting blameless retrospectives. It contrasts this full-scale testing approach with the scaled-model strategy, highlighting the former's benefits in identifying issues at scale and the latter's drawbacks.
  3. Planning in Bets: Risk Mitigation at Scale - ShopifyThis post introduces a four-question framework for risk mitigation: 1. What are the risks? (Visibility through WCGW exercises). 2. What is worth mitigating? (Optimizing time through voting and expert discussion). 3. Who makes what decisions? (Aligning incentives and accountability by empowering decision-makers). 4. How do you communicate? (Alignment and awareness through over-communication). It also frames engineering challenges as 'bets' on probabilities to deal with uncertainty, emphasizing the distinction between decision quality and outcomes.
  4. 10 Tips for Building Resilient Payment Systems - ShopifyThis post contributes 10 tips for building resilient payment systems, focusing on practical techniques and lessons learned from Shopify's experience at scale. Key contributions include: 1. Recommending low timeouts for HTTP clients and data stores (e.g., Net::HTTP, MySQL MAX_EXECUTION_TIME) as a starting point. 2. Advocating for circuit breakers (e.g., Semian) to prevent cascading failures and detailing their configuration nuances (e.g., fine-grained identifiers). 3. Explaining capacity management through queueing theory (Little's Law) and its implications for throughput and latency. 4. Outlining the four golden signals for monitoring (latency, traffic, errors, saturation) and the need for structured logging with correlation IDs. 5. Detailing the use of idempotency keys (ULIDs) for exactly-once processing of payment and refund operations, including recovery steps. 6. Highlighting the importance of consistent reconciliation for data integrity. 7. Emphasizing understanding system behavior under load and the role of rate limiting and load shedding. 8. Discussing the challenges of testing and simulating real-world traffic (e.g., BFCM 2021 load testing with Google Cloud).
  5. Cloud, Load, and Modular Code: What 2022 Looks Like for Shopify - ShopifyThis post details Shopify's technical preparations for BFCM 2021 and outlines future plans for 2022. It highlights key metrics from BFCM 2021, including peak app server requests (32M RPM), load balancer peaks (34M RPM), emails sent (145M), egress network traffic (30 TB/min), API calls (42B), webhooks delivered (13B), storefront caching data (3.18 PB written, 15 PB read), and MySQL queries (11M QPS). It also discusses the strategic initiatives for 2022: building more modular code (splitting storefront rendering, exploring majestic monoliths), leveraging cloud development platforms (Spin for faster environment setup), and making load testing a more natural part of the development process (automating and combining load tests). The post also mentions the YJIT compiler's impact and the continued commitment to Rails and React Native.
  7. Scaling Shopify's BFCM Live Map: An Apache Flink Redesign - ShopifyThis post details the redesign of the BFCM live map's data infrastructure using Apache Flink to address scalability issues and introduce new metrics. The previous Go-based streaming service (Cricket) was a bottleneck. The new Flink pipeline filters irrelevant checkout events, handles high volumes (up to 1 million events/sec in load tests), and processes new metrics like product trends and unique shoppers. The system achieved 100% uptime during BFCM with no manual intervention. A minor issue with Redis was quickly resolved. Future plans involve consolidating all processing logic within Flink to further simplify the system.
  8. Dynamic ProxySQL Query Rules - ShopifyIntroduced a tool for dynamically managing ProxySQL query rules across thousands of instances. Implemented a dry-run mode for query rules by leveraging the `active` and `log` fields in ProxySQL's `mysql_query_rules` table, allowing for safe previewing of rule effects. Developed a sidecar container that polls a key-value store for rule changes and applies them to ProxySQL, ensuring scalability and reducing manual effort during incidents. Incorporated exponential backoff for logging to prevent performance degradation from overly broad regex patterns.
  9. Resiliency Planning for High-Traffic Events - ShopifyThis post details the evolution of Shopify's resiliency planning for high-traffic events. It elaborates on the use of load and stress testing to identify system limits and regressions, and introduces the concept of a user-centric resiliency matrix to document failure modes and their impact on user experience. The post also emphasizes the importance of 'game days' to test the documented model against reality and adjust in a controlled setting, exercising technical, human, and organizational defenses. It highlights the need to avoid single root cause attribution in incident analysis and to address deeper systemic issues. For BFCM specifically, the post notes a slowdown in the rate of change, with increased focus on performance, reliability, and scale, including more frequent load tests and game days.
  10. Organizing 2000 Developers for BFCM in a Remote World - ShopifyThis post details the program management and communication strategies employed to coordinate over 2000 developers for BFCM. It outlines the process of understanding opportunities for improvement, building an effective program at scale, creating adjustments to the workflow specifically for BFCM, executing the program, and iterating for the next year. Key contributions include the implementation of a distributed ownership model for platform preparation tasks, a risk assessment process for PRs in critical areas using the 'Caution Tape' bot, and a phased deployment strategy with reduced PRs per deploy and extended canary periods. The post also highlights the communication plan, including Slack channels, GitHub repos, internal documentation, and presentations, to ensure company-wide alignment.
  11. Capacity Planning at Scale - ShopifyThis post details Shopify's approach to capacity planning for BFCM, emphasizing the collaboration with Google Cloud Platform (GCP) for resource forecasting. It outlines the process of creating a master resourcing plan, estimating resource needs (CPUs, storage) for key areas, and incorporating buffers for flexibility and regional failover. The post also highlights the importance of scalability testing, introducing the concept of "faux" BFCM events (e.g., Oktoberfest scale-up) to proactively identify and resolve bottlenecks before the actual event. It mentions the use of go-lua for load testing and the benefits of cloud elasticity for temporary resource allocation during these tests.
  12. Pummelling the Platform–Performance Testing Shopify - ShopifyThis post details Shopify's approach to performance testing for large sales events like BFCM. It introduces the concepts of load testing and stress testing, and explains the importance of generating realistic traffic shapes and hitting a variety of realistic endpoints. The post describes the internal load generation tool using Go and Lua (go-lua), and the development of Hardy Har Har (HHH) for HAR-based load testing to simulate real browser sessions at scale. It also highlights the use of ChatOps ('Spy') for test execution and reporting. The core contribution is the detailed explanation of the methodology and tooling used to ensure platform resilience under extreme load.
  13. Your Circuit Breaker is Misconfigured - ShopifyThis post details the configuration and parameter tuning of Semian Circuit Breaker, a Shopify-developed resiliency toolkit. It explains how to predict application behavior during failures and optimize worker utilization by carefully configuring parameters such as error_threshold, error_timeout, and half_open_resource_timeout. The post provides detailed explanations and diagrams for each parameter, illustrating their impact on system stability and worker utilization during service outages. It highlights a specific configuration change that reduced utilization requirements from 263% to 4%.
  14. Four Steps to Creating Effective Game Day Tests - ShopifyThis post details best practices for conducting Game Day tests to improve system and human resilience. It outlines a four-step process: 1. Listing all potential failure scenarios by collaborating with the team and considering infrastructure, network, and load-related issues. 2. Creating a series of controlled experiments with testable hypotheses about expected system behaviors and recovery mechanisms. 3. Testing human systems by analyzing team communication, work distribution, and coordination under pressure during incident response. 4. Addressing any gaps uncovered during the tests by analyzing root causes and planning corrective actions, followed by re-testing.
  15. Preparing Shopify for Black Friday and Cyber Monday - ShopifyThis post details Shopify's preparations for the 2018 Black Friday and Cyber Monday (BFCM) period, highlighting the importance of resiliency planning and operational readiness. Key contributions include: - Describing "game days" as a fault injection technique to test system assumptions by degrading dependencies (e.g., artificial latency to shipping providers) and verifying fallback mechanisms. - Outlining load testing exercises to simulate BFCM traffic volumes and ensure system behavior under stress. - Explaining the implementation of a feature freeze and code freeze to minimize changes and reduce the risk of introducing bugs during peak periods. - Quantifying the success of these preparations by reporting record-breaking sales and system metrics, emphasizing the payoff of proactive risk mitigation and understanding third-party dependencies.
  16. Surviving Flashes of High-Write Traffic Using Scriptable Load Balancers (Part II) - ShopifyThis post details the implementation of stateless fair queueing using PID controllers to manage high-write traffic during flash sales. It introduces a mechanism to assign priority based on user timestamps, using a dynamically calculated threshold managed by a proportional PID controller. This approach avoids a central data store, maintaining statelessness at the edge tier and significantly reducing queue time variance, as evidenced by improved P95, average, and median queue times.
  17. Surviving Flashes of High-Write Traffic Using Scriptable Load Balancers (Part I) - ShopifyThis post details the implementation of a scriptable load balancer using Nginx and Lua to handle extreme traffic spikes during flash sales. It addresses the challenges of checkout write traffic by implementing a leaky bucket algorithm and a cached queue page to maintain customer experience while protecting the platform. The solution was developed as a short-term measure to bridge the gap until a full refactor of the checkout flow could occur.
  18. Building and Testing Resilient Ruby on Rails Applications - ShopifyThis post details Shopify's efforts to improve system resiliency in preparation for high traffic events. It introduces Toxiproxy, a TCP proxy for simulating network conditions in integration tests, and discusses the application of circuit breakers and Semian to handle slow or unavailable external services. The post outlines a strategy for identifying service dependencies and creating matrices to test application behavior under degraded external service states. It also explains how decorators are used to provide fallbacks for unavailable resources and how Semian leverages connection pooling concepts to fail fast when acquiring external resources.