TechPrufer
Get started
Confluent
Confluent logoConfluent · Feature trail

Multi-Geo Replication

12 posts · Sep 2016 – Oct 2025

What this trail covers

Confluent Platform continues to evolve its multi-datacenter replication capabilities, now enabling secure, private cross-cloud data replication over Confluent's backbone. This enhances existing Cluster Linking functionality by supporting private networking between AWS, Azure, and Google Cloud, facilitating use cases like data sharing, disaster recovery, and aggregated analytics without traversing the public internet. The platform offers native enterprise-grade protections including IP filtering, and now supports Azure Private Link for secure, private data replication between Kafka clusters within private Azure networks.

All Confluent trails
  1. Secure Cross-Cloud Data Replication with ConfluentThis post details the enhancement of Confluent's Cluster Linking to support cross-cloud replication over private networks. It introduces the capability to replicate data between Confluent Cloud clusters in AWS, Azure, and Google Cloud using private networking constructs like PrivateLink and VNet peering. The post outlines the technical benefits, including reduced latency and data transfer costs, and provides a practical example using Terraform to provision Enterprise clusters and Confluent CLI commands to configure Cluster Linking for topic mirroring. It emphasizes the security aspects, including the use of dedicated API keys with least privilege and the avoidance of public internet traversal.
  2. Cross-Data-Center Apache Kafka® Replication: Decision Framework & Ops GuideThis post provides a decision framework and operational guide for cross-data-center Apache Kafka replication using Kafka MirrorMaker. It details replication patterns (Active-Passive vs. Active-Active, Within Region vs. Across Regions), recovery goals (RTO/RPO), and the operational readiness checklist for MirrorMaker 2. Key features of MirrorMaker 2, such as efficient data replication, multi-cluster replication, offset syncing, selective topic replication, and automatic failure recovery, are highlighted. Best practices for operators are also discussed, covering configuration issues, replication lag, memory/JVM crashes, and broker connectivity failures.
  3. Secure Inter-Organizational Data SharingThis post introduces Stream Sharing as a new, simpler method for inter-organizational data sharing, complementing existing Cluster Linking and direct Kafka/REST API integration. Stream Sharing allows secure, read-only sharing of single Kafka topics with external parties via email invitations, offering authenticated sharing, layered encryption, fine-grained access management, and instant revocation. It enforces schemas for data quality and compatibility. The post also elaborates on the technical challenges of inter-organizational data sharing from both producer (security) and recipient (reliability) perspectives, contrasting traditional methods like SFTP and REST APIs with the benefits of data streaming via Confluent.
  4. Cluster Linking for Azure Private Link is available in Confluent CloudIntroduces Cluster Linking support for Azure Private Link, enabling secure data replication between Kafka clusters in private Azure networks without traversing the public internet. This extends existing multi-cloud replication capabilities by allowing traffic to remain within the private network via Azure Private Link, reducing operational complexity compared to self-managed mirroring solutions, and enabling real-time, high-throughput data movement.
  5. Multi-Geo Replication in Apache KafkaThis post details the evolution of multi-geo replication for Apache Kafka, covering stretched clusters (synchronous replication across datacenters) and connected clusters (asynchronous replication using Kafka Connect or Cluster Linking). It explains Kafka's replication mechanisms (ISR, leader epochs, acks), introduces Confluent Server's observer replicas with automatic promotion for enhanced availability, and elaborates on Cluster Linking's ability to mirror topics, synchronize metadata (consumer offsets, ACLs), and maintain offset parity across clusters. It also touches upon Fetch From Followers (KIP-392) for consumer read optimization.
  7. Easy, Seamless Hybrid Cloud Cluster Linking with Confluent 7.1This post introduces Cluster Linking as a generally available feature in Confluent Platform 7.1 and Confluent Cloud. It details how Cluster Linking provides a secure, seamless, and built-in solution for hybrid cloud data replication by mirroring data, topic configuration, and consumer offsets between on-premises and cloud clusters. The post highlights key benefits such as seamless transitions for consumer applications, enhanced security through source-initiated connections, reliability in unstable network conditions, scalability, and ease of use via CLI/REST API commands. It provides a practical example of creating a source-initiated cluster link from on-premises to Confluent Cloud, including configuration details and commands for creating mirror topics.
  8. Confluent’s Newest Features in Q3 2021This post announces the general availability of fully managed Cluster Linking, a new geo-replication technology that directly connects independent Kafka clusters across regions and clouds. It highlights the simplification of data movement, increased reliability for global deployments, and unification of cloud environments. The post contrasts Cluster Linking with older replication tools like MirrorMaker 2 and Replicator, emphasizing its advantages in reducing infrastructure costs, operational burden, and architectural complexity. It details how Cluster Linking enables global data replication, regional disaster recovery, and simplified workload migrations by providing perfectly mirrored and globally consistent topic replication with offset preservation and support for non-Java consumers.
  9. Automatic Observer Promotion for Safe, Multi-Datacenter Failover in Confluent 6.1This post details the introduction of Automatic Observer Promotion in Confluent Platform 6.1. It explains the concept of observers as asynchronous replicas in Multi-Region Clusters, the limitations of manual failover (tediousness, potential data loss via unclean leader election) in previous versions, and the technical implementation of automatic promotion and demotion. The new `observerPromotionPolicy` field is introduced, with options like `under-min-isr` and `under-replicated`, to control the promotion behavior. The post also discusses the benefits for a 2.5 DC architecture.
  10. Introducing Cluster Linking in Confluent Platform 6.0Introduces Confluent Cluster Linking, a new feature for replicating topics between independent Kafka clusters. It allows for byte-for-byte, offset-preserving mirroring of topics, including metadata, consumer offsets, and ACLs. The feature enables seamless topic migration, cloud bursting, and disaster recovery by allowing the destination cluster to fetch data from the source. Mirror topics are managed independently and can be promoted to normal topics. The initial release supports destination-initiated links and will evolve to support source-initiated links for hybrid deployments.
  11. Project Metamorphosis Month 5: Global Event Streaming in Confluent CloudIntroduces Cluster Linking in Confluent Cloud, a new feature for global event streaming. Cluster Linking allows for seamless, offset-preserving replication of topics between independent Kafka clusters, creating a 'virtual Kafka cluster' spanning the globe. It aims to provide the same guarantees as internal Kafka protocols, including exact order, partitioning, and offsets. Use cases include hybrid cloud replication between on-premises Kafka and Confluent Cloud, and multi-cloud replication across different cloud providers. The initial version supports unidirectional transfers into Confluent Cloud or Confluent Server, with bidirectional replication planned for later. Future enhancements will include establishing links from either source or destination and reversing links for easier failover/failback.
  12. How Tencent PCG Scales Massive Data Pipelines with Apache KafkaThis post details Tencent PCG's development of a federated Kafka design using a proxy layer to scale massive data pipelines. They built producer and consumer proxies to map logical topics to physical topics across multiple Kafka clusters. A custom controller manages federated metadata, and a name service handles proxy broker discovery. This architecture allows for seamless capacity expansion, improved fault tolerance, and transparent physical cluster migration, handling trillions of messages daily with low metadata refresh latency. Limitations include potential out-of-order messages for keyed data and proxy interface evolution.
  13. 15 Things You Should Know About ReplicatorThis post introduces Confluent Replicator as a solution for multi-cluster Kafka data replication, complementing Multi-Region Clusters. It details 15 key features and benefits of Replicator, including enabling hybrid cloud architectures, out-of-the-box configuration and monitoring via Confluent Control Center, schema migration, RBAC support, and disaster recovery capabilities with offset translation. It also highlights its ability to aggregate messages from multiple clusters and its role in enabling new replication flows.
  14. Conquering Hybrid Cloud with Replicated Event-Driven ArchitecturesThis post details how Confluent Replicator enables hybrid cloud architectures by replicating event streams between on-premises Kafka clusters and Confluent Cloud. It highlights the benefits of using Kafka as a central event nervous system across environments, reducing security and complexity risks. The post uses a fintech use case (electronic stock trading) to illustrate migrating Tier 2 components to the cloud for cost optimization while keeping Tier 1 systems on-premises for low latency. It also introduces a demo for setting up a replicated event stream between Google Kubernetes Engine (GKE) and Confluent Cloud using Confluent Operator and Helm Charts.
  15. Multi-Region Clusters with Confluent Platform 5.4Introduces Multi-Region Clusters in Confluent Platform 5.4, built into Confluent Server. Details three key features: Follower Fetching (KIP-392) for consumers to read from the closest replica to reduce cross-WAN traffic and latency; Observers as asynchronous replicas that replicate data but do not join the ISR or become leaders, improving durability without impacting write throughput and preventing ISR thrashing; and a new JSON-based Replica Placement strategy for specifying replica counts and constraints across different 'racks' (locations). Also discusses multi-region ZooKeeper deployment considerations and KIP-500.
  16. Monitoring Data Replication in Multi-Datacenter Apache Kafka Deployments | ConfluentThis post details how Confluent Control Center can be used to manage and monitor multi-datacenter Apache Kafka deployments, specifically focusing on Confluent Replicator. It explains how to monitor Replicator's performance, consumer lag, and data synchronization status to ensure SLAs are met. The post also highlights the use of Confluent Monitoring Interceptors to gather client metadata for understanding data flow and client-topic mappings. A GitHub example for setting up an active-active multi-datacenter environment with bidirectional replication is provided.
  17. Apache Kafka® Disaster Recovery | Confluent Best PracticesThis post details the importance of multi-region disaster recovery for Apache Kafka deployments, especially for mission-critical workloads. It highlights Confluent Cloud features like Cluster Linking and Schema Linking for building multi-region, multi-cloud, and hybrid cloud deployments. The post also introduces Confluent Replicator as a key component for managing cross-datacenter replication and outlines essential building blocks for multi-datacenter designs, including data replication, timestamp preservation, preventing cyclic repetition, resetting consumer offsets, and centralized schema management. It emphasizes the need for a robust disaster recovery plan to mitigate downtime and data loss.
  18. Enterprise Streaming Multi-Datacenter Replication with Kafka | ConfluentThis post introduces Multi-Datacenter Replication as a new capability within Confluent Platform, designed to simplify and automate data replication between Kafka clusters across multiple datacenters. It addresses the limitations of MirrorMaker by providing dynamic topic creation with matching partition counts and replication factors, automatic topic resizing and reconfiguration, flexible topic selection, support for secure clusters, and integration with Confluent Connect API for scalability and fault tolerance. The implementation leverages a Kafka source connector and is configurable and monitorable via Confluent Control Center. The post details an aggregation use case and provides setup instructions.
  19. Announcing Confluent 3.1 with Apache Kafka 0.10.1.0 | ConfluentIntroduced Multi-datacenter Replication as a new capability within Confluent Platform, allowing for data and topic configuration replication between Kafka clusters in different data centers. This feature can be managed via Confluent Control Center or command-line tools.
  20. Use Apache Kafka to Migrate to AWS Cloud | ConfluentThis post introduces a strategy for gradual cloud migration using Apache Kafka and MirrorMaker. It proposes a central pipeline architecture where data from on-prem Kafka clusters is replicated to AWS Kafka clusters. This approach aims to reduce development and operational costs, improve data quality and consistency, and provide a long-term, multi-datacenter solution for organizations migrating to the cloud. It highlights the benefits of proven multi-datacenter architecture, continuous low-latency synchronization, centralized manageability, enhanced security and governance, and cost savings.