How to Reliably Scale Your Data Platform for High Volumes - Shopify

What this post added

This post details the approach taken by the Data Platform Engineering (DPE) team to scale Shopify's data platform for high-volume events like BFCM. It outlines a 5-step process: 1. Identifying primary objectives of services (e.g., data freshness SLO for ingestion, latency SLO for query serving). 2. Pinpointing service knobs and levers (e.g., job frequency, overlap interval, memory/CPU requests for Longboat; BigTable node pool size, ProxySQL connection pool/queue size for Reportify). 3. Running load tests to simulate extreme throughput and identify constraints. 4. Confirming and updating mitigation strategies with actionable alerts and clear instructions, assuming recovery by someone sleep-deprived. 5. Turning the knobs back and making informed decisions about scaling infrastructure or planning resiliency projects. The post also mentions the tiered services taxonomy (Tier 1-4) used for prioritizing reliability and infrastructure budgets.