Grab

Zero traffic cost for Kafka consumers

Jul 7, 2023Fabrice Harbulot
Zero traffic cost for Kafka consumers
SummaryAI, from the source post

This post details the implementation of fetching Kafka consumer data from the closest replica to reduce cross-AZ traffic and associated costs. It covers the Kafka upgrade process to version 3.1, configuration changes for brokers (using `broker.rack` and `replica.selector.class`) and consumers (updating the Go SDK with `client.rack`), and the observed impact of a 25% reduction in cross-AZ traffic. Caveats discussed include increased end-to-end latency, challenges with graceful broker isolation, and potential load skewage across AZs, with mitigation strategies like exposing AZ metrics.

More from the Kafka on Kubernetes Fault Tolerance trail