Confluent

Incremental Cooperative Rebalancing in Apache Kafka

Sep 24, 2019Konstantine Karantasis
Incremental Cooperative Rebalancing in Apache Kafka
SummaryAI, from the source post

Details the challenges of 'stop-the-world' rebalancing in Kafka clients (Consumer, Connect, Streams) and introduces Incremental Cooperative Rebalancing. Explains the 'incremental' and 'cooperative' aspects of the new protocol. Provides a detailed walkthrough of how Incremental Cooperative Rebalancing works in Kafka Connect with three scenarios: a new worker joining, an existing worker bouncing, and an existing worker leaving permanently, illustrating how tasks are redistributed with minimal interruption.

More from the Incremental Cooperative Rebalancing trail