Incremental Cooperative Rebalancing

2 posts · Aug 2019 – Jun 2025

What this trail covers

This feature thread tracks the evolution of Kafka's consumer rebalance protocol, moving from the classic eager and cooperative strategies to a new, server-driven protocol (KIP-848) in Kafka 4.0+. This new protocol aims to eliminate consumer downtime, reduce rebalance times, improve stability and scalability, and simplify management by shifting coordination logic to the broker-side group coordinator. It supports live upgrades and is designed to become the default in future Kafka releases. Future enhancements include client-side performance improvements and full topic ID support for offsets. Work is also underway to bring these benefits to Kafka Streams via KIP-1071.

All Confluent trails

Jun 2025KIP-848: A New Consumer Rebalance Protocol for Apache Kafka® 4.0This post introduces KIP-848, a new, optional consumer rebalance protocol for Apache Kafka 4.0+. It fundamentally redesigns the consumer-broker interaction for rebalancing by shifting coordination logic to the broker-side group coordinator. Key innovations include a continuous heartbeat mechanism, server-driven reconciliation, declarative state from consumers, and incremental rebalancing without a global stop-the-world pause. This results in reduced/eliminated consumer downtime, faster rebalances, improved stability and scalability, and simplified management. The post details the adoption process, including cluster and client upgrades and configuration changes, and notes that live upgrades are supported.
May 2020Cooperative Rebalancing in the Kafka Consumer, Streams & ksqlDBThis post details the existing eager rebalancing protocol in Kafka consumers, highlighting its 'stop-the-world' nature and drawbacks like downtime scaling with partition count. It then introduces the concept of an 'ideal' rebalancing protocol where only partitions needing to move are affected, minimizing downtime. The post lays the groundwork for the incremental cooperative rebalancing protocol by explaining the challenges of direct consumer-to-consumer communication and the need for a mechanism to indicate partition relinquishment.
Sep 2019Incremental Cooperative Rebalancing in Apache KafkaDetails the challenges of 'stop-the-world' rebalancing in Kafka clients (Consumer, Connect, Streams) and introduces Incremental Cooperative Rebalancing. Explains the 'incremental' and 'cooperative' aspects of the new protocol. Provides a detailed walkthrough of how Incremental Cooperative Rebalancing works in Kafka Connect with three scenarios: a new worker joining, an existing worker bouncing, and an existing worker leaving permanently, illustrating how tasks are redistributed with minimal interruption.
Aug 2019Kafka Connect Improvements in Apache Kafka 2.3 | ConfluentThis post details the introduction of Incremental Cooperative Rebalancing (KIP-415) in Apache Kafka 2.3 for Kafka Connect, which replaces the previous 'stop-the-world' rebalancing mechanism. It includes a practical demonstration using datagen and Elasticsearch connectors to visualize the reduction in downtime. Additionally, it highlights logging improvements (KIP-449) with Mapped Diagnostic Context (MDC) for better connector/task identification, REST API enhancements (KIP-465) for consolidated connector status and info retrieval, and the setting of distinct client.ids per task (KIP-411) for improved diagnostics. Finally, it introduces connector-level producer/consumer configuration overrides (KIP-458) via the `connector.client.config.override.policy` setting.
Advertisement