TechPrufer
Sign inGet started

Blogs›Confluent›Multi-Kafka Cluster Management with Cluster Registry

Multi-Kafka Cluster Management with Cluster Registry

Multi-Kafka Cluster Management with Cluster Registry

3
posts
2019–2021

This release details enhancements to Confluent Control Center for managing large-scale Kafka deployments. Key improvements include better visualization and dynamic modification of broker configurations without requiring restarts, addressing challenges in distributed systems. Additionally, Control Center's scalability has been significantly improved to manage a larger number of partition replicas, covering over 90% of existing customer workloads. This post complements the existing thread on managing multiple Kafka Connect and KSQL clusters by focusing on the core Kafka broker management aspects.

2021

Integrate Confluent Control Center with Cluster Registry for the Easiest Multi-Kafka Cluster Management

3/5/2021

Introduced support in Confluent Control Center (version 6.0.1) to read cluster information from Cluster Registry. This enables Control Center to use Cluster Registry as the primary source of cluster information, avoiding the need to manually configure each cluster in Control Center's properties file. Provided migration scripts (control-center-export and cluster-information-migration-script) to facilitate the transition from Control Center's configuration to Cluster Registry. Enabled updating cluster names in Control Center, with changes persisted in Cluster Registry.

2019

Dawn of Kafka DevOps: Managing Multi-Cluster Kafka Connect and KSQL with Control Center | Confluent

5/8/2019

This post introduces the management of multiple Kafka Connect and KSQL clusters using Confluent Control Center. It provides a practical example of setting up and managing separate clusters for data ingestion (Kafka Connect), data transformation (KSQL), and data streaming to a database (Kafka Connect), as well as for analytical purposes (KSQL). It highlights the benefits of resource isolation, environment separation, and application separation in multi-cluster deployments.

Dawn of Kafka DevOps: Managing Kafka Clusters at Scale with Control Center | Confluent

5/2/2019

This post introduces enhancements in Confluent Control Center 5.2 for managing large Kafka clusters. It details the ability to visualize differences in broker configurations and dynamically reconfigure brokers without restarts, leveraging KIP-226. The post also highlights improved scalability, enabling Control Center to manage approximately 120,000 partition replicas (around 40,000 partitions with a replication factor of three), which covers over 90% of customer workloads.