Apache Kafka 4.1 Release: New Features & Upgrade Guide

This post announces Apache Kafka 4.1.0, highlighting new features and upgrade paths. Key additions include: Queues for Kafka (preview), a new Streams Rebalance Protocol (early access) based on the consumer group protocol, and improvements to Kafka Broker, Controller, Producer, Consumer, and Admin Client. Notable KIPs include plugin metric registration, consistent transaction error handling, consumer close options for group leaving, rack topology change rebalancing, unified consumer topic metrics, deadlock protection for producer network threads, OAuth JWT-bearer grant type support, endpoint listener name cleanup, and transactional ID pattern filtering for ListTransactions API. Kafka Streams sees KIPs for moving window configurations to SerDes, the new Streams Rebalance Protocol, and enforcing explicit naming for internal topics. Kafka Connect gains support for running multiple versions of connector plugins and KIP-877 for plugin metric registration.