Kafka Connect Improvements in Apache Kafka 2.3 | Confluent

This post details the introduction of Incremental Cooperative Rebalancing (KIP-415) in Apache Kafka 2.3 for Kafka Connect, which replaces the previous 'stop-the-world' rebalancing mechanism. It includes a practical demonstration using datagen and Elasticsearch connectors to visualize the reduction in downtime. Additionally, it highlights logging improvements (KIP-449) with Mapped Diagnostic Context (MDC) for better connector/task identification, REST API enhancements (KIP-465) for consolidated connector status and info retrieval, and the setting of distinct client.ids per task (KIP-411) for improved diagnostics. Finally, it introduces connector-level producer/consumer configuration overrides (KIP-458) via the `connector.client.config.override.policy` setting.