Blogs›Confluent›Kafka Producer Partitioning Strategy

Kafka Producer Partitioning Strategy

posts

2019

This post introduces the sticky partitioning strategy for Apache Kafka producers, a new feature in version 2.4. This strategy aims to improve producer latency by optimizing batching, particularly for records with null keys. It addresses the inefficiency of the previous round-robin approach for null keys, which led to smaller batches and increased latency. The sticky partitioner selects a single partition for non-keyed records until the batch is full or a timeout occurs, then randomly selects a new partition. This results in larger batches and reduced latency, especially as the number of partitions increases. Performance tests using the Castle framework and Trogdor benchmarks demonstrate significant reductions in p99 latency and decreased CPU utilization compared to the default partitioning strategy. The post also explores the performance of the sticky partitioner with keyed messages and different linger.ms settings, showing consistent or improved latency.

2019

Apache Kafka Producer Improvements: Sticky Partitioner

12/18/2019

Introduces the sticky partitioning strategy for Apache Kafka producers, available from version 2.4. This strategy optimizes batching for records with null keys by selecting a single partition for a batch before randomly choosing a new one. It includes performance benchmarks demonstrating reduced latency and CPU usage compared to the previous default partitioning strategy, especially with increased partitions and null keys. The post also details the `onNewBatch` interface addition to the partitioner for implementing sticky partitioning and discusses its performance with keyed messages and varying `linger.ms` settings.