Optimally Scaling Kafka Consumer Applications

This post details Grab's optimization of Kafka consumer applications on Kubernetes. It covers the transition from Horizontal Pod Autoscaling (HPA) to Vertical Pod Autoscaling (VPA) with a fixed number of pods, resulting in a ~45% reduction in resource usage. It also discusses managing availability through Kubernetes Priority Classes to differentiate between latency-sensitive and latency-tolerant workloads, leveraging Spot instances for non-critical pipelines, and implementing overprovisioning with noop pods to ensure quick rescheduling of evicted pods and reduce deployment times.