Making Kafka Serverless: Lessons From Confluent Cloud

This post details the architecture and implementation of Confluent Cloud's serverless Kafka offering. It describes the four key components: the Confluent Cloud control plane, the Confluent Kubernetes Operator, Self-Balancing Clusters, and Infinite Storage. The control plane manages provisioning, scaling, and API requests, translating customer intents into Kubernetes custom resources (Physical Stateful Clusters - PSCs). The Confluent Kubernetes Operator consumes PSCs to manage Kafka clusters. Self-Balancing Clusters automatically manage broker placement and storage balancing. Infinite Storage leverages tiered storage to object storage for elastic capacity. The post also discusses the microservices architecture, event-driven communication via Kafka (mothership Kafka), and the use of Kubernetes for managing services.