Kafka Clusters and Cluster Management

This post addresses the decision-making process for creating new Kafka clusters, detailing physical limitations (e.g., 200,000 partitions per cluster, 4,000 partitions per broker) and operational concerns like data privacy, noisy neighbors, and chargeback. It argues for the benefits of a single cluster for real-time data correlation and introduces Confluent Cluster Linking and Schema Linking as mechanisms for inter-cluster data exchange. It also provides a federated governance model for operating large Kafka deployments, distinguishing responsibilities between shared services and line-of-business teams, with a focus on security, capacity assurance, and self-service operations, and suggests Confluent Cloud as a SaaS solution for simplified management.