Debugging Systems in the Cloud: MySQL, Kubernetes, and Cgroups - Shopify

This post details the investigation and resolution of a performance issue impacting MySQL instances running on Shopify's KateSQL platform, which is built on Google Cloud's Kubernetes Engine (GKE). The issue manifested as slow Pod creation times and degraded query performance. The investigation involved deep dives into Kubernetes Pod startup, MySQL initialization, and Linux kernel memory cgroup behavior. Key findings included a bug in the Linux kernel's `mem_cgroup_commit_charge` function, exacerbated by a potential cgroup leak, leading to slow memory allocations. The post outlines immediate mitigation strategies (node replacement), further investigation steps (resource analysis, strace, perf captures), the breakthrough discovery of the cgroup issue, and the eventual fix through GKE version upgrades (to 1.18) which included kernel patches addressing the bug.