Introducing Kafka-Kit: Tools for scaling Kafka | Datadog

What this post added

Introduced Kafka-Kit, a collection of tools for scaling Kafka. Key components are `topicmappr` for partition-to-broker mapping and `autothrottle` for replication throttling. `topicmappr` offers deterministic output, minimal movement broker replacements, rack-aware placement, and replication factor updates. It supports 'count' and 'storage' placement strategies, with 'count' focusing on even leadership and partition distribution, and 'storage' prioritizing even storage utilization using bin-packing principles. `topicmappr` also enhances broker-to-broker relationship maximization and provides detailed broker distribution reports.