TechPrufer
Get started
Grab
Grab logoGrab · Feature trail

Kafka on Kubernetes Fault Tolerance

7 posts · Jan 2020 – Dec 2023

What this trail covers

Grab's real-time data streaming platform (Coban) has enhanced the fault tolerance and cost efficiency of its Kafka deployments. This includes enabling Kafka consumers to fetch from the closest replica to reduce cross-AZ traffic, upgrading Kafka clusters to version 3.1 with a phased rollout, and implementing rack-awareness for brokers and consumers. The platform also addresses increased end-to-end latency, graceful broker isolation during rotation, and potential load skewage across AZs by exposin

All Grab trails
  1. Kafka on Kubernetes: Reloaded for fault toleranceThis post details the evolution of Grab's Kafka on Kubernetes fault tolerance. The initial design suffered from client connection errors, manual NLB reconfiguration needs, and static provisioning issues with NVMe instance store volumes. The solution involves integrating the AWS Node Termination Handler (NTH) to trigger graceful Kafka shutdowns via SIGTERM, minimizing client disruption. For manual instance terminations, the `terminate-instance-in-auto-scaling-group` AWS CLI command is used. AWS maintenance events are handled by cordoning/draining nodes and utilizing the Kubernetes Cluster Autoscaler to provision new nodes. Crucially, the use of Strimzi's KafkaTopic and KafkaCluster CRDs with dynamic provisioning for NVMe instance store volumes resolves the static provisioning problem, ensuring new PVs and PVCs are automatically created on replacement nodes. This makes the Kafka cluster resilient to broker terminations without manual intervention.
  2. Zero traffic cost for Kafka consumersThis post details the implementation of fetching Kafka consumer data from the closest replica to reduce cross-AZ traffic and associated costs. It covers the Kafka upgrade process to version 3.1, configuration changes for brokers (using `broker.rack` and `replica.selector.class`) and consumers (updating the Go SDK with `client.rack`), and the observed impact of a 25% reduction in cross-AZ traffic. Caveats discussed include increased end-to-end latency, challenges with graceful broker isolation, and potential load skewage across AZs, with mitigation strategies like exposing AZ metrics.
  3. Zero trust with KafkaThis post details the implementation of zero-trust security for Grab's Kafka clusters. Key technical contributions include: 1. **Mutual TLS (mTLS)** for authentication and encryption, chosen over OAuth/SCRAM for offline peer identity verification. 2. **Hashicorp Vault and its PKI engine** for dynamic generation of short-lived client and server certificates, mitigating compromise risks. 3. **Policy-Based Access Control (PBAC)** with **Open Policy Agent (OPA)** for granular authorization, granting access based on the principle of least privilege. 4. Integration with **Strimzi** (Kafka on Kubernetes operator) for managing mTLS and OPA integration. 5. Detailed descriptions of server and client authentication flows, including CA tree structure. 6. Client setup using a Golang SDK with default mTLS configuration, retry mechanisms, and automatic certificate renewal. 7. Authorization process involving OPA policies written in Rego, managed via GitLab CI and pushed to S3, with Kafka brokers caching authorization outcomes. 8. Self-service authorization for topic owners via Git repository permissions and JSON data sources.
  4. How Kafka Connect helps move data seamlesslyThis post details Grab's adoption and implementation of Kafka Connect for various data integration scenarios. It covers the use of Debezium connectors for CDC from MySQL, leveraging binlog streams for data changes and snapshots, and handling DB upgrades/outages by managing connector offsets. It also explains the deployment of MirrorMaker2 on Kafka Connect for message mirroring and offset translation, enabling disaster recovery and zero-downtime migrations. Furthermore, it describes the development of a custom Protobuf-to-JSON converter for ingesting data into Azure Event Hubs. The deployment architecture using Docker and Kubernetes, managed by Terraform, is also outlined.
  5. Exposing a Kafka Cluster via a VPC Endpoint ServiceThis post details the implementation of a VPC Endpoint Service to expose an Apache Kafka cluster from Grab's main VPC to the GrabKios VPC. It describes the architecture involving a Network Load Balancer (NLB) with dedicated listeners for each broker and AZ-specific CNAME records to optimize data transfer across Availability Zones. The setup ensures deterministic connections to Kafka brokers while enhancing security by avoiding VPC peering. Future improvements focus on scalability and multi-cloud compatibility.
  7. Optimally Scaling Kafka Consumer ApplicationsThis post details Grab's optimization of Kafka consumer applications on Kubernetes. It covers the transition from Horizontal Pod Autoscaling (HPA) to Vertical Pod Autoscaling (VPA) with a fixed number of pods, resulting in a ~45% reduction in resource usage. It also discusses managing availability through Kubernetes Priority Classes to differentiate between latency-sensitive and latency-tolerant workloads, leveraging Spot instances for non-critical pipelines, and implementing overprovisioning with noop pods to ensure quick rescheduling of evicted pods and reduce deployment times.
  8. Plumbing At ScaleThis post details the development and deployment of an event sourcing platform in Go, built on top of critical Kafka clusters. It introduces a stream processing framework with capabilities like deduplication, filtering, mapping, aggregation, windowing, and joining. The architecture leverages Kubernetes on AWS for deployment infrastructure, utilizing Horizontal Pod Autoscaler and Cluster Autoscaler for elasticity. ScyllaDB is used as an intermediate metastore for pipelines, and results are exposed via the GrabStats service. The post also outlines the anatomy of a stream processing pod, including triggers, runtime, and the pipeline plugin.