Elastic Kafka Clusters in Confluent Cloud - Project Metamorphosis

This post announces Project Metamorphosis, focusing on the 'Elasticity' trait of cloud-native data systems. It introduces usage-based billing for all Confluent Cloud products, allowing dynamic provisioning and de-provisioning of Dedicated clusters. It also announces self-service expansion and automatic data balancing for Dedicated Kafka clusters in Confluent Cloud. For on-prem, Confluent Platform will support automatic data balancing with Confluent Operator for Kubernetes-like elasticity. Contributions to Apache Kafka, specifically KIP-500 and KIP-595 for a Raft-based metadata quorum, are highlighted as foundational improvements for elasticity and scalability.