Building Self-driving Kafka clusters using open source components

This post details the development and operationalization of self-driving Kafka clusters at Slack. It covers the migration to Kafka 2.0.1, standardizing clusters under a single version managed by one team. Key components include Chef and Terraform for build, release, provisioning, and deployment, with an internal fork of Kafka for reproducible builds. Zookeeper provisioning was automated using Consul KV and ASGs. To address hot spotting and instability, partition counts were made multiples of broker counts, consumer counts were smoothed, and replication bandwidth was limited. Cruise Control was integrated for automated cluster rebalancing. Chaos engineering was employed to test failure modes, leading to the adoption of jumbo frames and the use of separate Zookeeper clusters per Kafka cluster. Self-serve tools like Kafka Manager and a Kafka offset exporter were deployed to provide visibility and simplify operations for consumer teams. The upgrade process for Kafka clusters was redesigned to involve cutting over to a new cluster rather than in-place upgrades.