Introducing Kafka-Kit: Tools for scaling Kafka | Datadog

This post introduces Kafka-Kit, a collection of tools for managing and scaling Kafka clusters. The primary tools are `topicmappr` for partition placement and `autothrottle` for replication auto-throttling. `topicmappr` offers deterministic output, minimal movement broker replacements, configurable rack-aware partition placement (by count or storage size), and replication factor updates. It also provides clear action summaries and detailed broker distribution reports. The 'count' strategy aims for even leadership and partition distribution, while the 'storage' strategy prioritizes even storage utilization using bin-packing algorithms and metrics fetched from Datadog API.