TechPrufer
Get started
Confluent
Confluent logoConfluent · Feature trail

Kafka KRaft Migration

6 posts · Aug 2015 – Jul 2026

What this trail covers

Apache Kafka 3.8.0 continues the transition to KRaft mode, with KIP-848 (Next Generation Consumer Rebalance Protocol) now in preview. Tiered Storage supports clusters with multiple log directories (JBOD), and JBOD in KRaft is no longer early access. Compression level can now be specified (KIP-390). The Log4J Appender is deprecated. Docker images for GraalVM native Kafka broker and official Docker image are introduced. KIP-1036 enhances RecordDeserializationException for better DLQ/logging. KIP-1037 allows WriteTxnMarkers API with Alter Cluster Permission. `offsets.commit.required.acks` is deprecated. Kafka Streams gains Shareable State Stores (KIP-813), customizable task assignment (KIP-924), and improved StateStore Iterator metrics (KIP-989). Kafka Connect enforces `tasks.max` (KIP-1004).

All Confluent trails
  1. Announcing Confluent Platform 8.3: Powerful Apache Flink® SQL operations, Easier KRaft Migrations, Expanded Monitoring and moreThis post introduces Confluent Platform 8.3, which includes enhanced capabilities for migrating to KRaft. Key contributions include simplified KRaft migrations for 2.5 Datacenter (2.5DC) deployments with a supported path for lightweight tiebreaker sites, and for multi-region clusters by deriving ZooKeeper endpoints using Kafka. Confluent for Kubernetes (CFK) 3.3.0 and later simplifies this process. The release also offers improved tools for validating readiness and expanding quorum capacity with minimal disruption.
  2. Apache Kafka 4.3 released: 25 KIPs, Kafka Streams, Share Group Controls, Broker Isolation, and moreApache Kafka 4.3 introduces 25 KIPs, including enhancements to broker, controller, producer, consumer, and admin client functionalities. Key additions include follower fetch from tiered offset, mechanisms to cordon brokers and log directories, configurations for group coordinator buffer sizes, alignment of num.partitions and default.replication.factor for topic creation, and configurations for KRaft fetch and snapshot byte sizes. It also corrects the default min.insync.replicas for the `__remote_log_metadata` topic, adds additional group configurations for share groups, introduces assignment epochs for consumer groups, and adds partition size percentage metrics for storage monitoring. Support for OAuth Client Assertion to client_credentials Grant Type is also added. Kafka Streams sees improvements in StateStore managed changelog offsets, making Bytes utils public API, tracking in-memory state store size, wiping local state on startup, extending ProcessingExceptionHandler for GlobalThread, storing headers in state stores, and DSL opt-in support for headers-aware state stores. Kafka Connect improves offset translation in RemoteClusterUtils, enhances discoverability of configurable components, and updates MirrorMaker metrics. Deprecations include streams-scala, group.coordinator.rebalance.protocols, and existing MirrorMaker metrics.
  3. Introducing Confluent Platform 8.2: Queues for Kafka & Flink SQLConfluent Platform 8.2 introduces 'Queues for Kafka' with native queue semantics, elastic consumer scaling, and per-message acknowledgment, addressing previous limitations in task-processing workloads on Kafka. Flink SQL reaches General Availability (GA) with enhanced declarative stream management through DDLs (CREATE TABLE, DROP TABLE, ALTER TABLE), changelogs, and shared compute pools for improved resource efficiency and cost optimization. Operational flexibility is increased with multi-Kubernetes cluster support and OpenShift certification. Confluent Private Cloud Gateway 1.2 introduces 'Intelligent Fencing and Unfencing' for granular client traffic control during migrations and disaster recovery, and 'Salted Challenge Response Authentication Mechanism (SCRAM) auth swap' for protocol-level credential translation. Non-Java client support is expanded. New features also include cross-context authorization checks in Schema Registry, native Dead Letter Queue (DLQ) support for Kafka Connect, and anchored punctuation for stream applications. Control Center 2.5 receives significant UI updates for managing Queues for Kafka, Flink SQL, shared compute pools, multi-Kubernetes environments, and savepoints, alongside a redesigned landing page for monitoring extensive Kafka fleets.
  4. Migrate From Hosted Kafka to Confluent Cloud in the KRaft EraThis post introduces Kafka Copy Paste (KCP), an open-source tool from Confluent that simplifies the migration of Kafka workloads to Confluent Cloud. KCP utilizes a four-step process: discover and plan (using `kcp discover` and `kcp scan`), provision (generating Terraform IaC with `kcp create-asset`), migrate data (topics, ACLs, schemas, connectors), and migrate clients. A key technology highlighted is Cluster Linking, which enables byte-for-byte mirroring of topics, preserving partitioning, ordering, and offsets, and supporting continuous replication with robust error handling. The post also mentions External Cluster Linking over AWS Private Link as a future enhancement to KCP. It reiterates the benefits of migrating to Confluent Cloud, including cost savings (40%-70%), reduced operational burden, and a 99.99% uptime SLA, contrasting it with the limitations of hosted Kafka services.
  5. Apache Kafka 4.1 Release: New Features & Upgrade GuideThis post announces Apache Kafka 4.1.0, highlighting several key improvements and new features. The most significant contributions are the preview of 'Queues for Kafka' (KIP-932) and the early access introduction of a new Streams Rebalance Protocol (KIP-1071) based on KIP-848. It also details improvements to Kafka Broker, Controller, Producer, Consumer, and Admin Client, including plugin metric registration (KIP-877), consistent transaction error handling (KIP-1050), consumer close options for group leaving (KIP-1092), rack topology change triggers for rebalance (KIP-1101), unified consumer topic metrics (KIP-1109), deadlock protection on producer network thread (KIP-1118), OAuth jwt-bearer grant type support (KIP-1139), and transactional ID pattern filtering for ListTransactions API (KIP-1152). For Kafka Streams, it moves window configurations to SerDes (KIP-1020) and enforces explicit naming for internal topics (KIP-1111). Kafka Connect gains support for running multiple versions of connector plugins (KIP-891).
  7. Confluent Platform 8.0: Client-side field level encryption (GA), and MoreThis post announces Confluent Platform 8.0, highlighting the General Availability of KRaft mode, which completely replaces ZooKeeper for Kafka metadata management. This change simplifies Kafka architecture, improves scalability to millions of partitions, and enables near-instant controller failovers. The post details how Confluent for Kubernetes and Confluent Ansible Playbooks support automated, zero-disruption migrations from ZooKeeper to KRaft. It also introduces the next-generation Confluent Control Center, now Prometheus-based, offering significant performance improvements in metrics freshness, partition support, and startup times. Integration of Confluent Manager for Apache Flink (CMF) within Control Center is also noted. A key security enhancement is the General Availability of Client-Side Field Level Encryption (CSFLE), which allows for granular encryption of individual message fields. Other updates include FlinkSQL in Open Preview and 'Queues for Kafka' with share groups in Early Access.
  8. Apache Kafka 4.0 Release: Default KRaft, Queues, Faster RebalancesThis post announces Apache Kafka 4.0, highlighting key advancements. The primary contribution is the default enablement of KRaft mode, eliminating ZooKeeper dependency and simplifying operations. It introduces KIP-848 for a new, server-driven consumer rebalance protocol that significantly reduces downtime and latency. Early access to Queues for Kafka (KIP-932) is also featured, extending Kafka's capabilities to support traditional queue semantics. The release mandates Java 11 for clients/Streams and Java 17 for brokers/Connect/Tools, and removes message formats v0 and v1. New features like Eligible Leader Replicas (KIP-996) and Pre-Vote (KIP-996) enhance fault tolerance and reduce leader election disruptions. Enhancements to administration tools for new group types (KIP-1043, KIP-1099) and client rebootstrap mechanisms (KIP-1102) improve operability. Kafka Streams gains KIP-1104 for flexible foreign key extraction in KTable joins and KIP-1112 for custom processor wrapping.
  9. Apache Kafka 3.9 Release: Dynamic KRaft Quorums, Better ZK MigrationApache Kafka 3.9 introduces dynamic KRaft quorums via KIP-853, allowing for dynamic addition and removal of controller nodes using `kafka-metadata-quorum.sh` or AdminClient API, a significant improvement over static configurations that required downtime. This release also marks the final iteration of ZooKeeper migration tooling, with all feature gaps closed, making Kafka 3.9 a critical 'bridge release' for users needing to migrate from ZooKeeper mode to KRaft mode before Kafka 4.0 deprecates ZooKeeper support entirely. Tiered Storage is now production-ready with new features including dynamic disablement per topic (KIP-950), tiered storage quotas (KIP-956), and exposed earliest local and tiered offsets (KIP-1005, KIP-1057). Kafka Streams improvements include `log.summary.interval.ms` configuration (KIP-1049) and enhanced exception handling (KIP-1033). Kafka Connect enhancements include better null value handling in transformations (KIP-1040), offset translation control in MirrorSourceConnector (KIP-1031), and a new health check endpoint (KIP-1017).
  10. Migrate from ZooKeeper to KRaft with Confluent for KubernetesThis post details Confluent's experience migrating thousands of Kafka clusters to KRaft in Confluent Cloud without downtime, highlighting the technical challenges of scale and multi-tenancy. It emphasizes KRaft's benefits in enhanced scalability (millions of partitions) and improved operational stability due to the removal of ZooKeeper. The post also outlines migration strategies for users, including manual steps (dual-write mode, rolling restarts) and automated approaches using Confluent for Kubernetes (CFK) and Ansible Playbooks.
  11. AK 3.8 Adds 17 KIPs: New Docker Images and much moreApache Kafka 3.8.0 introduces several key enhancements and deprecations. Notable additions include: KIP-390 for specifying compression levels, KIP-848 (preview) for a next-generation consumer rebalance protocol, KIP-899 for client rebootstrap capability, KIP-974 for GraalVM-based native Kafka broker Docker images, KIP-1018 for configuring remote fetch timeouts, KIP-1019 to expose metric measurability, KIP-1028 for the official Docker image, KIP-1036 to extend RecordDeserializationException, KIP-1037 to allow WriteTxnMarkers API with Alter Cluster Permission, and KIP-1047 to introduce a new Decoder API. Deprecations include Log4J Appender (KIP-719) and `offsets.commit.required.acks` (KIP-1041). Kafka Streams gains Shareable State Stores (KIP-813), customizable task assignment (KIP-924), and improved StateStore Iterator metrics (KIP-989). Kafka Connect enforces `tasks.max` (KIP-1004).
  12. Introducing Apache Kafka 3.7This post announces Apache Kafka 3.7.0, highlighting several key engineering advancements. It introduces early access support for JBOD disk failure handling in KRaft clusters (KIP-858), a significant step towards full KRaft parity. Deprecations for client APIs older than 2.1 (KIP-896) and Java 11 broker support (KIP-1013) are noted, signaling future removal. New features include broker-side client metrics and observability (KIP-714, KIP-1000), an optimized consumer rebalance protocol (KIP-848), and improved leader discovery for clients (KIP-951). An official Docker image (KIP-975), exponential backoff for client retries (KIP-580), and additional metrics for Tiered Storage (KIP-963) are also introduced. KRaft process management is improved (KIP-979), and transaction defense is enhanced (KIP-890). AdminClient can now directly communicate with KRaft controllers (KIP-919), and dynamic certificate reloading with different DN/SANs is supported (KIP-978). New metrics for KRaft performance (KIP-938) and controller ID (KIP-1001) are added. For Kafka Streams, rack-aware task assignment (KIP-925), expanded default DSL store configuration (KIP-954), relaxed non-null key requirements for joins (KIP-962), and a standby update listener (KIP-988) are introduced. Interactive query support for versioned state stores (KIP-960, KIP-968) and reverse queries for KV stores (KIP-985) are also included.
  13. What Is an Apache Kafka Cluster?This post provides a foundational explanation of Apache Kafka as a distributed computing system, detailing the roles of clusters, brokers, and partitions in achieving speed, durability, and scalability. It elaborates on how brokers manage load balancing and replication through partitions and leader/follower roles. A significant portion is dedicated to explaining the failure scenarios for followers and leaders, introducing KRaft as the mechanism for leader election and metadata management, and highlighting its role in replacing ZooKeeper.
  14. Why ZooKeeper Was Replaced with KRaft - The Log of All Logs | Confluent Why Replace ZooKeeper with Kafka Raft - The Log of All LogsThis post details the rationale for replacing ZooKeeper with KRaft for Kafka metadata management. It explains the scalability limitations of the old controller design, particularly during broker shutdowns and controller failovers, and the inherent challenges of using ZooKeeper as a metadata store. The post then introduces KRaft as a solution, explaining the concept of a metadata log, the benefits of using quorum replication (Raft algorithm) over primary-backup, and the leader election protocol within KRaft.
  15. 42 Ways ZooKeeper Removal Improves Apache KafkaThis post details the benefits of removing ZooKeeper from Apache Kafka, focusing on the simplification of administration, capacity planning, performance, monitoring, and troubleshooting. It highlights specific tasks that can be stopped, such as operating a separate distributed system, administering additional servers, managing separate security configurations, and dealing with ZooKeeper-specific tuning parameters and maintenance. The post also emphasizes performance improvements due to local metadata storage in brokers and the elimination of O(N) costs for topic operations. It provides concrete examples of how client and tool configurations will change with the removal of ZooKeeper.
  16. Prepare Clients and Tools for Apache Kafka's Zookeeper Removal (KIP-500)This post details the technical steps and rationale for preparing Kafka clients and tools for the removal of ZooKeeper dependency (KIP-500). It outlines how to migrate configurations from `zookeeper.connect` to `bootstrap.servers` for client applications, Confluent Platform components, and administrative tools. It also covers updating REST Proxy API calls from v1 to v2/v3 and methods for obtaining the Kafka cluster ID without relying on ZooKeeper. The post emphasizes the security benefits of connecting directly to brokers and the importance of proactive auditing and code upgrades.
  17. Kafka Needs No Keeper - Removing ZooKeeper DependencyThis post introduces KIP-500, a proposal to remove ZooKeeper dependency from Apache Kafka by storing metadata within Kafka itself using a Raft quorum. It details the problems with external metadata management (duplication, complexity, inefficiency, scalability limits) and outlines the proposed solution: treating metadata as a log, with standby controllers ready to take over. The post also discusses the roadmap for removing ZooKeeper from administrative tools, implementing self-managed metadata quorums via KIP-595, enabling KIP-500 mode, and the strategy for zero-downtime upgrades using bridge releases.
  18. How Unity Built a Massive Data Infrastructure with Confluent PlatformThis post details Unity's successful migration of petabytes of data and operations from AWS to GCP using Confluent Replicator. It highlights the use of Confluent Platform and Apache Kafka for their data infrastructure, handling approximately half a million events per second with peaks of a million events per second, and achieving zero outages resulting in money loss. The migration was completed over a few months with Confluent Professional Services' guidance on Kafka architecture and best practices, ensuring safety and speed to mitigate doubled cloud costs. The post also touches upon the shift from batch to event streaming models, reducing data latency from two days to 15 minutes, and the improved reliability of Kafka on Confluent Platform, leading to teams building more event streaming systems and exploring its use for faster machine learning model training.
  19. Log Compaction | Highlights in the Apache Kafka Community - Feb 2017This post discusses several Kafka Improvement Proposals (KIPs) that were active in February 2017. Key contributions include KIP-112 and KIP-113 for handling disk failures and replica movement in JBOD configurations, KIP-117 for a public AdminClient API, KIP-98 for exactly-once delivery and transactional messaging, KIP-118 for dropping Java 7 support in version 0.11, KIP-110 for adding ZStandard compression, and KIP-109 for deprecating old consumers. It also highlights external blog posts on microservice caching with Kafka APIs, Kafka for monolith-to-microservices transition, Kafka-centric streaming architecture, compression codec benchmarks, and using Kafka for machine learning systems.
  20. Log Compaction | Highlights in the Apache Kafka and Stream Processing Community | November 2016 | ConfluentThis post details the rationale for KIP-72 (limiting memory for incoming requests) and KIP-81 (bounding fetch memory usage in consumers), both addressing broker and consumer memory exhaustion. It also discusses KIP-84 (SASL/SCRAM mechanisms) for improved authentication and KIP-85 (dynamic JAAS configuration) for client security in containerized environments. Additionally, KIP-87 (Compaction Tombstone Flag) is introduced as a more robust way to delete keys in compacted topics compared to null values.
  21. Distributed Consensus Reloaded: Apache ZooKeeper and Replication in Apache Kafka | ConfluentThis post explains the fundamental problem of distributed consensus and how systems like ZooKeeper solve it by implementing atomic broadcast. It details how Kafka leverages ZooKeeper for metadata management, including broker sets, leader election, and eligible leader subsets. The post also highlights potential pitfalls in reasoning about consensus and agreement, using an example of workers consuming different flavors due to timing differences in ZooKeeper reads. It contrasts ZooKeeper's role as a 'coordination kernel' with explicit consensus APIs and explains how Kafka uses ZooKeeper's sequential nodes and state updates to achieve agreement on leader replicas.