Running Apache Kafka on Kubernetes at Shopify - Shopify

What this post added

This post details Shopify's migration of its Apache Kafka infrastructure from on-premises data centers to Google Cloud Platform (GCP). It outlines the three-step migration process, the decision to use Kubernetes (specifically StatefulSets and Custom Resources) over Virtual Machines for managing Kafka clusters, and best practices for containerizing Kafka and ensuring cluster availability. Key practices include node affinity, taints, inter-pod anti-affinity, persistent volumes, rack awareness, and readiness probes.