Incremental Cooperative Rebalancing in Apache Kafka
Sep 24, 2019Konstantine Karantasis

Summary
Details the challenges of 'stop-the-world' rebalancing in Kafka clients (Consumer, Connect, Streams) and introduces Incremental Cooperative Rebalancing. Explains the 'incremental' and 'cooperative' aspects of the new protocol. Provides a detailed walkthrough of how Incremental Cooperative Rebalancing works in Kafka Connect with three scenarios: a new worker joining, an existing worker bouncing, and an existing worker leaving permanently, illustrating how tasks are redistributed with minimal interruption.