Database Sharding and Pod Architecture

Shopify has introduced database sharding to horizontally scale its databases. To address the resilience issues arising from sharding, they developed a 'pods' architecture. Each pod is an isolated unit containing a set of shops and their dedicated datastores, along with shared resources like job workers and app servers. Crucially, there is no cross-pod communication, ensuring that a failure in one pod does not impact others. A 'Sorting Hat' component in load balancers routes requests to the appropriate pod. This post details how composite primary keys can mitigate deadlocks in high concurrency environments when updating data in sharded databases, by ensuring that gap locks required for updates remain within each process instance, dramatically reducing the chances of overlapping gaps.