Zero traffic cost for Kafka consumers
Jul 7, 2023Fabrice Harbulot

Summary
This post details the implementation of fetching Kafka consumer data from the closest replica to reduce cross-AZ traffic and associated costs. It covers the Kafka upgrade process to version 3.1, configuration changes for brokers (using `broker.rack` and `replica.selector.class`) and consumers (updating the Go SDK with `client.rack`), and the observed impact of a 25% reduction in cross-AZ traffic. Caveats discussed include increased end-to-end latency, challenges with graceful broker isolation, and potential load skewage across AZs, with mitigation strategies like exposing AZ metrics.