TechPrufer
Sign inGet started

Blogs›Confluent›Kafka KRaft Migration

Kafka KRaft Migration

Kafka KRaft Migration

5
posts
2019–2024

This post details Unity's successful migration of thousands of Kafka clusters from AWS to GCP, involving petabytes of data, using Confluent Replicator for mirroring traffic and Confluent Platform for operational stability. It highlights the shift from batch processing to event streaming, reducing latency from two days to 15 minutes, and improved reliability leading to Kafka not being a suspect for issues. The post also mentions plans for feeding machine learning models with real-time data and using Ansible Playbooks for upgrades.

2024

Migrate from ZooKeeper to KRaft with Confluent for Kubernetes

10/14/2024

This post describes Confluent's experience migrating thousands of Kafka clusters from ZooKeeper to KRaft, detailing the technical challenges of handling high-throughput, multi-tenant clusters and the benefits of enhanced scalability and operational stability. It outlines the migration process, including dual-write mode and rolling restarts, and introduces automated migration tools like Confluent for Kubernetes (CFK) and Ansible Playbooks.

2022

Why ZooKeeper Was Replaced with KRaft - The Log of All Logs | Confluent Why Replace ZooKeeper with Kafka Raft - The Log of All Logs

3/22/2022

This post delves into the technical reasons for replacing ZooKeeper with KRaft for Kafka metadata management. It explains the scalability limitations of the old controller design with ZooKeeper, particularly during broker shutdowns and controller failovers, detailing the impact of ZooKeeper's read/write load and Znode limitations. It then introduces KRaft as a solution, explaining its implementation as a metadata log using the Raft consensus algorithm for quorum replication, and the necessity of a new leader election protocol within KRaft to overcome ZooKeeper's limitations.

2021

42 Ways ZooKeeper Removal Improves Apache Kafka

2/18/2021

This post details the benefits of removing ZooKeeper from Apache Kafka, focusing on 42 specific areas where operational complexity is reduced. It covers improvements in administration, capacity planning, disk usage, performance, monitoring, and troubleshooting. The post also provides guidance on preparing clients and tools for the transition to a ZooKeeper-less Kafka, including changes to connection strings and administrative tool usage.

2020

Prepare Clients and Tools for Apache Kafka's Zookeeper Removal (KIP-500)

10/27/2020

This post details how to prepare Kafka clients, administrative tools, and the REST Proxy API for the removal of ZooKeeper dependency (KIP-500). It provides specific configuration changes (e.g., `bootstrap.servers` instead of `zookeeper.connect`), command examples for administrative tools (e.g., `kafka-topics --bootstrap-server`), and API version updates for the REST Proxy. It also covers methods for retrieving the Kafka cluster ID without relying on ZooKeeper.

2019

How Unity Built a Massive Data Infrastructure with Confluent Platform

11/19/2019

Unity migrated its massive data infrastructure from AWS to GCP, handling half a million to a million events per second, using Confluent Platform and Kafka. They employed Confluent Replicator for a safe, zero-downtime, zero-data-loss migration of petabytes of data. This transition enabled a move from batch processing to event streaming, reducing data latency significantly and improving the perceived stability of Kafka, leading to increased adoption for new event streaming systems and plans for ML model integration.