TechPrufer
Get started
Confluent
Confluent logoConfluent · Feature trail

Kafka Request Purgatory Performance

2 posts · Oct 2015

What this trail covers

This feature thread tracks the evolution of Kafka's request purgatory, focusing on performance and memory efficiency. The initial implementation using Java's DelayQueue suffered from O(log n) insert/delete costs and inefficient purging, leading to potential OutOfMemoryErrors and performance degradation under high load. The new design leverages Hierarchical Timing Wheels and doubly linked lists to achieve O(1) insert/delete operations for completed requests and a more efficient purging mechanism. Benchmarks demonstrate significant improvements in request per second (RPS) throughput and reduced CPU usage, especially in high enqueue rate scenarios. The new design also optimizes the driving clock using DelayQueue by enqueuing task buckets instead of individual tasks, further reducing overhead.

All Confluent trails
  1. Apache Kafka, Purgatory, and Hierarchical Timing Wheels | ConfluentThis post details the redesign of Kafka's request purgatory to address performance and memory issues. It explains the deficiencies of the old design, which used Java's DelayQueue and suffered from inefficient purging, leading to potential OutOfMemoryErrors and performance bottlenecks. The new design introduces Hierarchical Timing Wheels and doubly linked lists to achieve O(1) insert/delete for completed requests and a more efficient purging strategy. It also optimizes the driving clock by enqueuing task buckets instead of individual tasks into DelayQueue. Benchmark results show a dramatic improvement in enqueue performance, with the new implementation handling significantly higher RPS and exhibiting better CPU usage compared to the old design, especially under high load and memory-constrained conditions.