TechPrufer
Get started
Datadog
Datadog logoDatadog · Feature trail

Kafka Scaling Tools

1 post · Aug 2018 – Feb 2025

What this trail covers

Datadog's Streaming Platform provides a control layer for managing massive, multi-cluster Kafka deployments. It abstracts Kafka complexity with 'Streams' for resilient, multi-cluster pipelines, a custom 'Assigner' for real-time coordination and failovers, and 'Stream lanes' for QoS and dead-letter handling. The platform also introduces an advanced commit log to overcome head-of-line blocking and a custom Kafka client, 'libstreaming', for performance and observability. This enables dynamic workload management and real-time reliability at scale.

All Datadog trails
  1. Achieving relentless Kafka reliability at scale with the Streaming Platform | DatadogThis post details the evolution of Datadog's Streaming Platform for managing large-scale Kafka deployments. Key advancements include: 1. Streams: abstracting Kafka complexity for resilient, multi-cluster pipelines with stable identifiers, enabling real-time traffic failovers and dynamic reconfiguration. 2. Consumer Semantics: Shifting to an at-least-once delivery model and relaxing ordering guarantees at the processing stage to enable massive parallelism and re-establishing order downstream. 3. Assigner: A custom, real-time multi-cluster coordinator that monitors cluster health and workload distribution for rapid failovers and intelligent load balancing, reacting in seconds rather than minutes. 4. Stream lanes: Introducing independent lanes within a Stream for different priority levels (QoS) to isolate traffic and prevent interference, including a dedicated dead-letter queue (DLQ) lane for poison pills. 5. Advanced Commit Log: Enhancing Kafka's commit log to track multiple offsets or ranges of offsets within a partition simultaneously, allowing consumers to process live traffic while reprocessing backlogs or retrying failed events. 6. libstreaming: A custom Kafka client built for optimized performance and observability across all applications.
  2. Introducing Kafka-Kit: Tools for scaling Kafka | DatadogThis post introduces Kafka-Kit, a collection of tools for managing and scaling Kafka clusters. The primary tools are `topicmappr` for partition placement and `autothrottle` for replication auto-throttling. `topicmappr` offers deterministic output, minimal movement broker replacements, configurable rack-aware partition placement (by count or storage size), and replication factor updates. It also provides clear action summaries and detailed broker distribution reports. The 'count' strategy aims for even leadership and partition distribution, while the 'storage' strategy prioritizes even storage utilization using bin-packing algorithms and metrics fetched from Datadog API.