Running Apache Kafka on Kubernetes at Shopify - Shopify

This post details Shopify's migration of its Apache Kafka infrastructure from on-premises data centers to Google Cloud Platform (GCP) and its subsequent deployment on Kubernetes. It outlines the three-step migration process, the rationale for choosing Kubernetes over VMs (specifically the use of StatefulSets and Custom Resources), best practices for containerizing Kafka (building own images, resource monitoring containers), and key configurations for cluster availability (Node Affinity, Taints, Inter-pod Affinity/Anti-Affinity, Persistent Volumes, Custom Resources for topic provisioning, rack-awareness, and Readiness Probes).