TechPrufer
Sign inGet started

Blogs›Confluent›Incremental Cooperative Rebalancing

Incremental Cooperative Rebalancing

Incremental Cooperative Rebalancing

2
posts
2019

Introduces Incremental Cooperative Rebalancing for Kafka clients (Consumer, Connect, Streams) to improve load balancing efficiency and reduce disruption during rebalances. This new protocol allows for staged convergence to a balanced state and avoids 'stop-the-world' rebalancing, particularly beneficial for large-scale deployments, multi-tenancy, and dynamic environments like Kubernetes. Kafka Connect is the first client to implement this, treating tasks as lightweight threads for more flexible rebalancing. Also includes logging improvements using Mapped Diagnostic Context (MDC) for better context in worker logs, consolidated REST API endpoints for connector status and info, distinct client.id per task for easier diagnostics, and connector-level producer/consumer configuration overrides.

2019

Incremental Cooperative Rebalancing in Apache Kafka

9/24/2019

Details the challenges of the existing 'stop-the-world' rebalancing protocol in Kafka clients, such as scaling issues, multi-tenancy disruptions, and Kubernetes node failures. Introduces the concept of Incremental Cooperative Rebalancing, emphasizing its incremental and cooperative nature. Explains the implementation in Kafka Connect, illustrating scenarios like a new worker joining, an existing worker bouncing, and an existing worker leaving permanently, highlighting how unaffected tasks continue to run and how rebalancing delays are utilized for graceful recovery.

Kafka Connect Improvements in Apache Kafka 2.3 | Confluent

8/15/2019

Details the implementation and benefits of Incremental Cooperative Rebalancing (KIP-415) in Kafka Connect starting from Apache Kafka 2.3, which replaces the previous 'stop-the-world' rebalancing with a more efficient, incremental approach. It also introduces logging improvements via Mapped Diagnostic Context (MDC) for better log context, consolidated REST API endpoints for connector status and information, distinct client.id per task for improved diagnostics, and the ability to override producer/consumer configurations at the connector level (KIP-458). Includes practical examples and test results demonstrating the reduction in downtime during rebalances.