Webserver Connection Handling and Request Dropping

2 posts · Mar 2013 – May 2026

What this trail covers

Shopify has evolved its approach to managing webserver connection handling and request dropping mechanisms to improve application resilience under high load. This includes developing dynamic query rules for ProxySQL to reroute, rewrite, or reject database queries, and implementing a tool to manage these rules at scale across thousands of instances. The system incorporates a dry-run mode for query rules to prevent accidental data corruption or downtime, and employs exponential backoff for logging.

All Shopify trails

May 2026We replaced Redis with MySQL for inventory reservations—and it scaled (2026) - ShopifyThis post details the migration of Shopify's inventory reservation system from Redis to MySQL. The core technical challenge was to maintain high-throughput, ACID-compliant inventory reservations at scale. The solution involved a new database design using one row per sellable unit, leveraging MySQL 8's `SKIP LOCKED` feature, and implementing a bounded pool of available rows with inline replenishment. Key technical decisions included using a composite primary key to reduce lock contention, switching to `READ COMMITTED` isolation to avoid gap locks, standardizing lock ordering to prevent deadlocks, and batching queries with `UNION ALL`. A significant finding was that the real bottleneck was not CPU or query performance, but connection exhaustion. The team implemented per-caller attribution of connection hold times by tagging SQL statements at the application layer and aggregating this data at the ProxySQL layer. This revealed that other checkout processes were holding connections longer than necessary, impacting reservation throughput. Optimizations included cleaning up these other checkout paths and re-evaluating MySQL's InnoDB thread concurrency configuration. The cutover was performed in shadow mode, running both Redis and MySQL in parallel before switching the source of truth to MySQL.
Nov 2025Run for your money: Engineering a treadmill that prints store credit (2025) - ShopifyThis post details the engineering effort behind a gamified treadmill experience that rewards runners with in-store credit, integrated with a custom POS UI extension. It covers the hardware setup (manual treadmill, Runn sensor, TrueForm Trainer, POS hardware, receipt printer, displays, server), software architecture (Python script for FTMS data, Remix app for game logic and state management, POS UI extension for discount application), and the logic for translating runner pace into earned credit. It also describes the state machine for game control and the process of printing receipts with scannable barcodes for discount redemption. The POS UI extension handles fetching user data, adding customers to the cart, and applying discounts.
Jul 2021High Availability by Offloading Work Into the Background - ShopifyThis post introduces the concept of offloading heavy lifting tasks into background jobs to improve application availability and response times. It details how background jobs work by using message queues to decouple the application server from worker processes. The benefits discussed include improved response times, spikeability, retries and redundancy, parallelization, prioritization, event-based and time-based scheduling, and simplicity of code. It also outlines challenges such as breaking changes to job parameters, lack of exactly-once delivery, and non-transactional queuing, suggesting the outbox pattern as a solution for transactional staging.
Apr 2021Dynamic ProxySQL Query Rules - ShopifyThis post details the development of a tool to dynamically manage ProxySQL query rules across thousands of instances. The tool uses a web app to define rules, which are then stored in a key-value store. A sidecar container running alongside ProxySQL polls this store and applies rule changes. The post also introduces a 'dry run' mode for query rules, implemented by modifying the `active` and `log` fields in ProxySQL's `mysql_query_rules` table, and discusses mitigating logging degradation with exponential backoff.
Mar 2013What Does Your Webserver Do When a User Hits Refresh? - ShopifyThis post details a specific patch to the Unicorn webserver that adds a `check_client_connection` configuration option. This option allows Unicorn to write a small amount of data to the client socket before rendering a response. If the write fails (indicating a disconnected client), the request is dropped, preventing wasted application server resources. This directly addresses the problem of HTTP 499 responses and the snowball effect during flash sales.
Advertisement