TechPrufer
Get started
Confluent
Confluent logoConfluent · Feature trail

Multi-Kafka Cluster Management with Cluster Registry

3 posts · May 2019 – Oct 2025

What this trail covers

Unified Stream Manager (USM) is now generally available with Confluent Platform 8.1, enabling a unified view for governance and monitoring of all Confluent clusters, whether on-prem, in a private cloud, or across public clouds. USM connects on-prem Confluent Platform environments to Confluent Cloud through a secure private network connection (initially AWS PrivateLink), sharing only metadata and telemetry. This provides a single source of truth for schemas, enforcing data contracts and enabling client-side field-level encryption (CSFLE) across hybrid infrastructure. It also consolidates observability, allowing users to monitor the health and performance of all clusters from the Confluent Cloud UI, trace data flow end-to-end, and discover data in a unified catalog. The architecture keeps message data within the private environment, sending only operational metadata to Confluent Cloud.

All Confluent trails
  1. Unified Stream Manager GA in Confluent Platform 8.1This post announces the general availability of Unified Stream Manager (USM) with Confluent Platform 8.1. USM enables unified governance and monitoring of hybrid and multi-cloud Confluent environments by connecting on-prem Confluent Platform deployments to Confluent Cloud via a secure private network connection (AWS PrivateLink). It centralizes schema management by making Confluent Cloud Schema Registry the single source of truth, with on-prem Schema Registries acting as read-only caches. It also introduces client-side field-level encryption (CSFLE) for sensitive data. For observability, USM consolidates monitoring of on-prem clusters within the Confluent Cloud UI, provides a unified data catalog via Data Portal, and enables end-to-end data stream tracing. The USM agent, deployed on-prem, securely streams metadata and telemetry to Confluent Cloud, ensuring client data remains local. The release also highlights enhancements in Control Center, Queues for Kafka (Open Preview), and Azure User Assigned Managed Identity (UAMI) for Kafka OAuth authentication. Confluent Platform for Apache Flink® updates include support for Flink 2.0.0/2.1.0, REST API management of savepoints, PyFlink and TableAPI support, and external metadata persistence in PostgreSQL or SQL Server.
  2. Next Generation of Control Center for Confluent PlatformIntroduces the next generation of Confluent Control Center (2.0) with a new Prometheus architecture for metrics collection and storage, enabling support for up to 400K partitions, reducing startup time to 1 minute, and achieving metrics freshness within 2-3 minutes. This eliminates the need for a separate Kafka cluster for metrics, simplifying operations and reducing costs.
  3. Integrate Confluent Control Center with Cluster Registry for the Easiest Multi-Kafka Cluster ManagementIntroduces integration between Confluent Control Center and Cluster Registry in Confluent Platform 6.0.1. This allows Control Center to read cluster information from Cluster Registry, eliminating duplicate work and potential errors from manual registration in both systems. It also adds migration scripts to facilitate moving existing Control Center configurations to Cluster Registry and allows updating cluster names directly in Control Center, which are then persisted in Cluster Registry.
  4. Dawn of Kafka DevOps: Managing Multi-Cluster Kafka Connect and KSQL with Control Center | ConfluentThis post introduces the management of multiple Kafka Connect and KSQL clusters using Confluent Control Center. It provides a practical example of setting up and managing separate clusters for data ingestion (Kafka Connect), data transformation (KSQL), and data streaming to a database (Kafka Connect), as well as for analytical purposes (KSQL). It highlights the benefits of resource isolation, environment separation, and application separation in multi-cluster deployments.
  5. Dawn of Kafka DevOps: Managing Kafka Clusters at Scale with Control Center | ConfluentThis post introduces enhancements in Confluent Control Center 5.2 for managing large Kafka clusters. It details the ability to visualize differences in broker configurations and dynamically reconfigure brokers without restarts, leveraging KIP-226. The post also highlights improved scalability, enabling Control Center to manage approximately 120,000 partition replicas (around 40,000 partitions with a replication factor of three), which covers over 90% of customer workloads.