Kafka on Kubernetes: Reloaded for fault tolerance

This post details the evolution of Grab's Kafka on Kubernetes fault tolerance. The initial design suffered from client connection errors, manual NLB reconfiguration needs, and static provisioning issues with NVMe instance store volumes. The solution involves integrating the AWS Node Termination Handler (NTH) to trigger graceful Kafka shutdowns via SIGTERM, minimizing client disruption. For manual instance terminations, the `terminate-instance-in-auto-scaling-group` AWS CLI command is used. AWS maintenance events are handled by cordoning/draining nodes and utilizing the Kubernetes Cluster Autoscaler to provision new nodes. Crucially, the use of Strimzi's KafkaTopic and KafkaCluster CRDs with dynamic provisioning for NVMe instance store volumes resolves the static provisioning problem, ensuring new PVs and PVCs are automatically created on replacement nodes. This makes the Kafka cluster resilient to broker terminations without manual intervention.