TechPrufer
Sign inGet started

Blogs›Datadog›Kafka Scaling Tools

Kafka Scaling Tools

Kafka Scaling Tools

1
posts
2018

Datadog developed Kafka-Kit, a suite of tools to manage and scale Kafka clusters. Kafka-Kit includes `topicmappr` for deterministic partition reassignments, broker replacements, rack-aware placement, and replication factor updates, supporting both 'count' and 'storage' placement strategies. It also includes `autothrottle` for replication auto-throttling. The tools aim to simplify operations for large-scale Kafka deployments, addressing challenges like data movement, capacity planning, and uneven storage utilization.

2018

Introducing Kafka-Kit: Tools for scaling Kafka | Datadog

8/13/2018

Introduced Kafka-Kit, a collection of tools for scaling Kafka. Key components are `topicmappr` for partition-to-broker mapping and `autothrottle` for replication throttling. `topicmappr` offers deterministic output, minimal movement broker replacements, rack-aware placement, and replication factor updates. It supports 'count' and 'storage' placement strategies, with 'count' focusing on even leadership and partition distribution, and 'storage' prioritizing even storage utilization using bin-packing principles. `topicmappr` also enhances broker-to-broker relationship maximization and provides detailed broker distribution reports.