Resiliency Planning and Game Days

11/20/2025

This post details the comprehensive BFCM 2025 readiness program, emphasizing year-round resilience building through capacity planning, infrastructure roadmapping, and risk assessments. It elaborates on the execution of Game Days, focusing on critical user journeys and cross-system simulations, and the development of the Resiliency Matrix. The post also details the load testing process using Genghis and Toxiproxy, outlining the three options for addressing identified limits (horizontal scaling, vertical scaling, optimizations). A significant portion is dedicated to new analytics challenges for systems launched post-BFCM 2024, including specific Game Day findings and mitigation strategies for Kafka partition increases, API layer memory optimization, and connection timeout tuning. Finally, it describes the scale testing program, detailing its methodology, types of tests, simulated user behaviors, critical scenarios, and the iterative identification and fixing of issues across multiple test cycles, including a mid-program shift to include authenticated checkout scenarios.

1/27/2023

This post details Shopify's approach to BFCM Scale Testing, a critical exercise for preparing the e-commerce platform for peak traffic. It outlines a five-step process: scaling up core platform and subsystems, running load tests simulating BFCM traffic, scaling down, analyzing results, and iterating on improvements. The post elaborates on the scaling-up strategy, including coordinating with cloud vendors and scaling upstream/downstream components. It describes five distinct load test flows (browsing/buying, admin, flash sale, Storefront API, Hydrogen/Oxygen) executed by the Cronograma tool. The methodology for determining test load, including using projections and testing regional failover, is explained. The importance of scaling down in reverse order of scale-up to protect dependencies is highlighted. Finally, the post discusses the blameless retrospective process for analyzing results and iterating on the plan, contrasting full-scale testing with scaled-model testing and emphasizing the benefits of the former for uncovering complex scaling issues.

12/6/2022

This post details a high-level risk mitigation process applied to large-scale systems, focusing on identifying risks through 'what could go wrong' exercises, prioritizing mitigation efforts based on business importance, defining decision-making processes by empowering subject matter experts, and ensuring alignment and awareness through communication. It introduces the concept of making 'bets' based on the probability of outcomes to deal with uncertainty in engineering.

4/20/2021

Introduced a tool for dynamically managing ProxySQL query rules across thousands of instances. Implemented a dry-run mode for query rules by leveraging the `active` and `log` fields in ProxySQL's `mysql_query_rules` table, allowing for safe previewing of rule effects. Developed a sidecar container that polls a key-value store for rule changes and applies them to ProxySQL, ensuring scalability and reducing manual effort during incidents. Incorporated exponential backoff for logging to prevent performance degradation from overly broad regex patterns.

12/10/2020

This post details Shopify's approach to resiliency planning for high-traffic events. It elaborates on the use of load and stress testing to identify regressions, the creation of user-centric resiliency matrices to understand failure modes and their impact on user experience, and the execution of game days to validate these models against reality. The post also emphasizes the importance of incident analysis for continuous improvement and outlines specific strategies for BFCM, such as reducing the rate of change and increasing testing frequency. It references 'How Complex Systems Fail' as a guiding principle.

12/7/2020

This post details the program management and communication strategies employed to organize over 2000 developers for BFCM preparation in a remote environment. It outlines the creation of a communication plan calendar, the use of a distributed ownership model with departmental champions, and the implementation of a 'Caution Tape' bot for risk assessment of PRs in critical repositories. It also describes strategies for slowing down the rate of production deployments, including reducing PRs per deploy, increasing canary testing duration, and locking deploys around the BFCM weekend. The post highlights the success of the program in achieving high PR review rates and enabling developers to focus on cleanup tasks and strategic planning, while also identifying areas for improvement in balancing developer productivity and platform stability.

12/3/2020

This post details Shopify's approach to capacity planning for BFCM, including forecasting traffic scenarios with data scientists, creating master resourcing plans for GCP, and tasking resource owners with detailed estimates. It highlights the challenges of planning during uncertain times (e.g., COVID-19) and the decision to scale to aggressive growth scenarios to ensure platform stability. The post also elaborates on the evolution of scalability testing, from historical surprises to regular "faux" BFCM events like "Oktoberfest scale-up", and the use of internal load testing tooling (go-lua) to identify and fix bottlenecks before the actual event.

12/2/2020

This post details the evolution of Shopify's performance testing strategy, moving from basic load and stress testing to more realistic simulations. It introduces the "Platform Conditioning" team's mission to improve tooling, processes, and culture around performance testing. The post describes the internal load generation tool using Go and Lua (go-lua VM) and the use of ChatOps ('Spy') for triggering tests. A significant contribution is the development of "Hardy Har Har" (HHH), a HAR-based load testing solution that replays actual browser network requests to simulate realistic user behavior at scale, overcoming the limitations of simple scripted load generation and the cost overhead of browser-based testing.

2/18/2020

This post details the configuration and behavior of the Semian Circuit Breaker, a Shopify-developed resiliency toolkit for Ruby. It explains the purpose of circuit breakers in preventing cascading failures during service outages and elaborates on the critical parameters: `name`, `error_threshold`, `error_timeout`, `half_open_resource_timeout`, and `success_threshold`. The post provides detailed explanations and diagrams illustrating how misconfigurations of these parameters can lead to severe system degradation and how proper tuning can significantly improve resilience and maintain worker utilization during failures. It emphasizes the importance of understanding the interplay between these parameters to achieve optimal system behavior under stress.

11/12/2019

This post details best practices for creating and executing Game Day tests. It outlines a four-step process: 1. Listing all potential failure scenarios by collaborating with the team and stretching imagination, then whittling down the list based on impact vs. simulation difficulty. 2. Creating a series of controlled experiments by expressing expectations as testable hypotheses, detailing prerequisite steps, expected behaviors, and restoration steps in a spreadsheet. 3. Testing human systems by analyzing team communication and work distribution under pressure. 4. Addressing any gaps uncovered by annotating experiment results, determining root causes, and planning corrections, followed by a follow-up discussion and updates to incident response guidelines. It also provides additional resources for further reading on systemic failure, human coordination, and resilience engineering.

12/21/2018

This post details Shopify's preparations for Black Friday and Cyber Monday (BFCM), focusing on Production Engineering's role in ensuring platform reliability and scalability. It highlights the use of 'game days' for fault injection (e.g., introducing artificial latency to shipping provider interactions) and load testing to simulate peak traffic. The post also discusses the implementation of feature and code freezes to minimize changes during the critical period. The success of these preparations is evidenced by record-breaking sales and order volumes during BFCM 2018, reinforcing the value of proactive risk management and dependency testing.

2/5/2017

This post details the implementation of stateless fair queueing using PID controllers to manage high-write traffic during flash sales. It introduces a mechanism to assign priority based on user timestamps, using a dynamically calculated threshold managed by a proportional PID controller. This approach avoids a central data store, maintaining statelessness at the edge tier and significantly reducing queue time variance, as evidenced by improved P95, average, and median queue times.

2/3/2017

This post details the implementation of a scriptable load balancer using Nginx and Lua to handle extreme traffic spikes during flash sales. It addresses the challenges of checkout write traffic by implementing a leaky bucket algorithm and a cached queue page to maintain customer experience while protecting the platform. The solution was developed as a short-term measure to bridge the gap until a full refactor of the checkout flow could occur.

1/29/2015

This post details the development and application of Toxiproxy, a TCP proxy for simulating network conditions in integration tests, and Semian, a tool inspired by Hystrix for managing external resource acquisition across processes using SysV semaphores. It also discusses the implementation of the circuit breaker pattern to handle slow or failing resources and the use of decorators to provide fallbacks for unavailable external services. The post emphasizes the importance of these tools in preparing for high-traffic events like Black Friday and Cyber Monday by improving system resiliency and removing single points of failure.