TechPrufer
Get started
Shopify
Shopify logoShopify · Feature trail

Container Resource Management with Systemd and CGroups

3 posts · Oct 2014 – Jul 2022

What this trail covers

Shopify has evolved its approach to container resource management, specifically addressing issues with systemd processes within containers not adhering to Kubernetes resource limits due to cgroup driver configurations. Initial efforts focused on containerizing applications and managing secrets. Subsequent development has focused on container resource management, addressing issues with systemd processes within containers not adhering to Kubernetes resource limits due to cgroup driver configurations. This post details a deep dive into a specific Linux kernel memory cgroup controller bug that caused inconsistent MySQL Pod startup times in KateSQL, leading to performance degradation. The investigation involved analyzing system calls, perf flamegraphs, and cgroup counts, ultimately identifying a cgroup leak. Mitigation strategies included node replacement and a workaround using `memory.force_empty`. The fix was confirmed by upgrading to GKE 1.18, which included upstream fixes for cgroup slab re-parenting bugs.

All Shopify trails
  1. Spin Infrastructure Adventures: Containers, Systemd, and CGroups - ShopifyThis post details the investigation and resolution of a cgroup leak issue where systemd processes, when run inside containers via `spin shell`, were not correctly placed within the `kubepods` cgroup hierarchy, leading to unhonored resource limitations. The issue was reproduced using `stress` and analyzed by comparing `kubectl exec` behavior with `spin shell` behavior, observing differences in cgroup assignments and resource usage. The root cause was identified as the `cgroupfs` cgroup driver used by default in Google's Container Optimized OS (COS), which does not properly delegate cgroup management to the host systemd. The fix involves the upcoming default switch to the `systemd` cgroup driver in COS version 101.
  2. The Story Behind Shopify’s Isospin Tooling - ShopifyThis post introduces Isospin, Shopify's systemd-based tooling for running applications within the Spin cloud development platform. It details how Isospin leverages systemd's unit management, template unit files, and generators to dynamically configure services, manage dependencies, and isolate processes within a single Linux VM. The post also describes additional tooling developed to address port collisions using a hashing process and to improve service readiness by utilizing systemd's notify socket mechanism.
  3. Debugging Systems in the Cloud: MySQL, Kubernetes, and Cgroups - ShopifyThis post details a deep dive into a specific Linux kernel memory cgroup controller bug that caused inconsistent MySQL Pod startup times in KateSQL, leading to performance degradation. The investigation involved analyzing system calls (`mmap()`, `mem_cgroup_commit_charge`), perf flamegraphs, and cgroup counts, ultimately identifying a cgroup leak. Mitigation strategies included node replacement and a workaround using `memory.force_empty`. The fix was confirmed by upgrading to GKE 1.18, which included upstream fixes for cgroup slab re-parenting bugs.
  4. Building an Internal Cloud with Docker and CoreOS - ShopifyThis post introduces Shopify's adoption of Docker and CoreOS to build an internal cloud for scaling and managing applications. It highlights the benefits of containers (portability, density, consistency, developer/ops workflows) and outlines future deep-dives into containerizing applications, managing secrets, routing, monitoring, bulletproofing, and provisioning. The post emphasizes the shift in responsibility towards developers owning containers and ops focusing on infrastructure.