Multi-Geo Replication

2/27/2023

This post details the technical aspects of multi-geo replication in Apache Kafka. It explains Kafka's core replication mechanisms, including partitions, replicas, leaders, followers, ISRs, and the role of offsets and high watermarks. It elaborates on producer durability configurations (`acks=0`, `acks=1`, `acks=all`) and the `min.insync.replicas` setting. The post contrasts stretched clusters (single Kafka cluster across datacenters) with connected clusters (multiple independent clusters). For stretched clusters, it discusses the benefits and limitations, and introduces Confluent Server's observer replicas with Automatic Observer Promotion for improved failover. For connected clusters, it details Confluent's Cluster Linking, explaining how it extends the replica fetcher protocol for cross-cluster replication, maintains offset parity, and synchronizes metadata like topic configurations, consumer offsets, and ACLs. It also briefly mentions Confluent Replicator as an alternative using Kafka Connect.

1/5/2022

This post introduces Confluent Cluster Linking, a new built-in capability for mirroring data, topic configuration, and consumer offsets between Confluent clusters in real-time. It details its application in hybrid cloud scenarios, emphasizing secure, seamless, reliable, scalable, and easy-to-use data bridging between on-premises and cloud deployments. The post provides specific CLI commands and configuration examples for creating source-initiated cluster links, including security credential management and mirror topic creation. It highlights the benefits over traditional tools like MirrorMaker 2 by eliminating extra infrastructure and de/recompression of messages.

8/17/2021

This post announces the general availability of Confluent Cloud Cluster Linking, a new feature for geo-replication and multi-cloud data movement. It simplifies connecting independent Kafka clusters across regions and clouds, enabling increased reliability of global Kafka deployments and unification of cloud environments. It highlights that Cluster Linking is a next-generation geo-replication technology that eliminates legacy middle-system approaches, allowing direct, bidirectional, and consistent replication between two clusters. It also mentions that consumer application offsets are mirrored, and provides benefits such as global data replication, regional disaster recovery, and accelerated project delivery times.

2/11/2021

Introduces Automatic Observer Promotion for Kafka observers, allowing them to be automatically promoted to the ISR list when min.insync.replicas is not met. Details the behavior, configuration options (observerPromotionPolicy: under-min-isr, under-replicated, leader-is-observer), and the technical benefits of zero RPO and low RTO compared to manual unclean leader election. Activates this feature by changing the replica placement policy version to two.

10/5/2020

Introduces Confluent Cluster Linking as a new capability for replicating topics between independent Kafka clusters. Details the mechanism of creating a cluster link and mirror topics, emphasizing offset preservation, real-time continuous replication, and independent ISR management. Highlights use cases for topic migration, cloud bursting, and disaster recovery. Explains the 'under the hood' architecture where the destination cluster fetches data from the source. Mentions future roadmap items like hybrid deployment support and improved UX.

8/24/2020

Introduces Cluster Linking in Confluent Cloud, a new feature for global event streaming. Cluster Linking allows Kafka clusters to be linked to create a single virtual global cluster, preserving Kafka's internal guarantees (order, partitioning, offsets) across linked clusters. It supports hybrid cloud replication (on-premises to Confluent Cloud) and multi-cloud replication (between different cloud providers). The post details use cases for hybrid cloud and multi-cloud strategies, and outlines future enhancements for networking and cross-cluster high availability. It also mentions that Cluster Linking is in early access on selected regions of Confluent Cloud and will be in preview in Confluent Platform 6.0.

8/3/2020

This post details Tencent PCG's development of a federated Kafka design using a proxy layer to scale massive data pipelines. They built producer and consumer proxies to map logical topics to physical topics across multiple Kafka clusters. A custom controller manages federated metadata, and a name service handles proxy broker discovery. This architecture allows for seamless capacity expansion, improved fault tolerance, and transparent physical cluster migration, handling trillions of messages daily with low metadata refresh latency. Limitations include potential out-of-order messages for keyed data and proxy interface evolution.

3/17/2020

This post introduces Confluent Replicator, a Kafka Connect connector for copying topic data between separate clusters. It details its capabilities for hybrid cloud architectures, bidirectional replication, and fan-out scenarios. The post highlights integration with Confluent Control Center for configuration and monitoring, schema migration between Schema Registry clusters, and RBAC for security. It also explains offset translation for disaster recovery and the aggregation of messages from multiple clusters into a single destination.

11/26/2019

This post details the use of Confluent Replicator to establish replicated event-driven architectures across hybrid cloud environments, specifically demonstrating a fintech use case for electronic stock trading. It explains how Replicator enables phased migrations from on-premises Kafka to Confluent Cloud, ensuring data durability and reducing operational complexity. The post also introduces a demo for setting up a GKE to Confluent Cloud replicated environment using Kubernetes and Confluent Operator.

9/16/2019

Introduces Multi-Region Clusters in Confluent Platform 5.4, enabling per-region data replication. Details Follower Fetching (KIP-392) allowing consumers to fetch from the closest replica to reduce cross-WAN traffic and latency. Explains Observers as asynchronous replicas that don't impact ISR or leadership, improving durability without affecting write throughput or causing ISR thrashing. Introduces a JSON-based Replica Placement strategy for specifying replica counts and constraints across regions, complementing Observers for disaster recovery. Discusses multi-region ZooKeeper deployment challenges and KIP-500.

4/10/2019

This post details how Confluent Control Center can be used to monitor Confluent Replicator for data replication in multi-datacenter Apache Kafka deployments. It explains how to track replication lag, throughput, and latency, and how Confluent Monitoring Interceptors provide insights into client application performance. The post also highlights the use of Control Center for managing Replicator instances and their configurations, and provides a demo for setting up and monitoring a multi-datacenter environment.

12/8/2016

Introduces Multi-Datacenter Replication as a new Confluent Platform capability to automate data copying between Kafka clusters in different datacenters. Highlights limitations of MirrorMaker and details the features of the new solution: dynamic topic creation with matching partition counts, replication factors, and topic configurations; automatic topic resizing and reconfiguration; topic selection via whitelists, blacklists, and regex; support for secure Kafka clusters; scalability and fault tolerance via Kafka Connect API; and integration with Confluent Control Center. Explains the implementation using a Kafka source connector and Kafka Connect, and provides an example of an aggregation use case.