Apache Kafka Producer Improvements: Sticky Partitioner

This post introduces and details the sticky partitioning strategy for Apache Kafka producers, available from version 2.4. It explains how this strategy addresses the latency issues associated with the previous round-robin approach for null-keyed records by selecting a single partition for non-keyed records until the batch is filled or a timeout occurs. The post provides performance benchmarks using the Castle test harness and Trogdor framework, demonstrating significant reductions in p99 latency and CPU usage, especially as the number of partitions increases. It also analyzes the performance impact on mixed key scenarios and confirms no significant latency increase for keyed messages.