TechPrufer
Sign inGet started

Blogs›Confluent›Kafka Request Purgatory Performance

Kafka Request Purgatory Performance

Kafka Request Purgatory Performance

2
posts
2015

This release details the redesign of Kafka's request purgatory to improve performance and reduce memory usage. The old implementation using DelayQueue had deficiencies in handling large numbers of asynchronous requests, leading to potential OutOfMemoryErrors and performance degradation due to frequent purging. The new design leverages Hierarchical Timing Wheels and doubly linked lists for O(1) insert/delete operations, significantly improving throughput and reducing CPU load. Benchmarks show a dramatic improvement.

2015

Log Compaction | Highlights in the Kafka and Stream Processing Community | November 2015 | Confluent

11/1/2015

This post highlights community efforts in Kafka development, including the addition of SASL/Kerberos-based authentication for the 0.9.0.0 release, client-side partition-assignment support in the new Kafka Consumer, and improvements to the Copycat export/import feature with distributed mode and a REST API. It also notes the replacement of old system tests with modular Ducktape-based integration tests. A key technical detail is the deep dive into Kafka's delayed response mechanism, referencing a separate blog post on the redesign of Kafka's request purgatory using Hierarchical Timing Wheels and doubly linked lists for improved performance and reduced memory usage.

Apache Kafka, Purgatory, and Hierarchical Timing Wheels | Confluent

10/28/2015

Introduced a new implementation for Kafka's request purgatory using Hierarchical Timing Wheels and doubly linked lists. This redesign addresses memory exhaustion and performance bottlenecks of the previous DelayQueue-based implementation by enabling immediate deletion of completed requests and reducing the overhead of the purge process. Benchmarks demonstrate substantial improvements in request throughput and CPU efficiency.