Confluent for Kubernetes

12 posts · Jul 2015 – Oct 2025

What this trail covers

Confluent for Kubernetes (CFK) simplifies running Kafka as a cloud-native data system on Kubernetes, enabling programmatic provisioning of Kafka resources with compute, storage, networking, and security options. It provides a consistent operational experience across on-prem and cloud deployments. This thread also tracks the evolution of tiered storage for decoupling Kafka's storage and compute layers, and self-balancing clusters for automated partition rebalancing and optimized resource utilization.

All Confluent trails

Oct 2025Introducing Confluent Private CloudThis post introduces Confluent Private Cloud, a new deployment model for platform teams. It highlights the Confluent Private Cloud Gateway, which acts as a smart, protocol-aware proxy between clients and Kafka clusters to centralize authentication, authorization, and encryption without requiring client code changes. It also introduces Intelligent Replication, a push-based replication mode that works alongside Kafka's pull-based approach to improve performance and latency. Finally, it mentions Unified Stream Manager (USM) as an add-on for unified management and monitoring across hybrid and multi-cloud Kafka deployments.
May 2021Introducing Confluent for KubernetesThis post announces the general availability of Confluent for Kubernetes (CFK). It details how CFK provides a cloud-native experience for self-managing Confluent Platform on private infrastructure, mirroring the capabilities of Confluent Cloud. Key technical contributions include the use of Kubernetes CustomResourceDefinitions (CRDs) and controllers for declarative API-driven management of Confluent services and data plane resources. The post highlights automated scaling, secure deployment configurations, rolling upgrades, and fault-tolerant infrastructure management through declarative specifications. It also emphasizes the adoption of GitOps principles for managing infrastructure as code and the integration with the broader Kubernetes ecosystem.
Sep 2020DevOps for Apache KafkaThis post introduces the 'streaming-ops' example project, which demonstrates modern DevOps practices for managing Apache Kafka on Kubernetes. It explains the benefits of running streaming applications on Kubernetes, the declarative resource management pattern, and the GitOps methodology using Flux. It also introduces the Operator pattern and how it applies to managing Confluent Platform and Kafka Connect. The post details the streaming-ops project's architecture, its use of Flux for GitOps workflows, and its approach to managing graduated environments using Kustomize.
Sep 2020Deploy Kafka on Kubernetes in Minutes with Confluent and OpenShiftThis post details the process of deploying Confluent Platform on OpenShift Container Platform (OCP) using the Confluent Operator. It outlines the prerequisites for both OCP and Confluent Operator installation, including AWS account setup, Red Hat network access, and the OCP SDK. The post walks through configuring OCP, deploying the Confluent Operator, and then deploying Confluent Platform components. It also touches upon choosing persistent storage for Kafka on both on-premises and public cloud Kubernetes clusters.
May 2020Elastically Auto Scaling Kafka on Kubernetes with ConfluentThis post introduces the upcoming features in Confluent Platform that enable elastic scaling of self-managed Kafka on Kubernetes: the Kubernetes Operator (already available), Tiered Storage (in preview), and Self-Balancing Clusters (coming soon). The Kubernetes Operator simplifies programmatic provisioning and deployment of Kafka brokers. Tiered Storage decouples compute and storage, allowing independent scaling and faster rebalancing by leveraging object storage. Self-Balancing Clusters will automate partition rebalancing when new brokers are added. The synergy of these features aims to make dynamic scaling fast and easy, reducing manual steps and operational burden.
Advertisement
May 2020Elastic Kafka Clusters in Confluent Cloud - Project MetamorphosisThis post introduces Project Metamorphosis, focusing on the 'Elasticity' trait. It details Confluent Cloud's move to a single usage-based billing model for all products, including Dedicated clusters, allowing dynamic provisioning and scaling. It also announces self-service expansion and automatic data balancing for Dedicated clusters. For on-prem, Confluent Platform will support automatic data balancing with Confluent Operator for Kubernetes. Contributions to Apache Kafka (KIP-500 and KIP-595) are highlighted to remove ZooKeeper dependency and enable scaling to millions of partitions.
Oct 2019How to Deploy & Run Confluent Platform on Pivotal Container Service (PKS)This tutorial details the deployment of Confluent Platform on Pivotal Container Service (PKS) using Confluent Operator. It covers the setup of Kafka and ZooKeeper clusters on PKS, configuration for external endpoint exposure, and verification steps for both internal and external connectivity. It also briefly mentions the deployment of Schema Registry.
Oct 2018Apache Kafka on Kubernetes – Could You? Should You? | ConfluentThis post discusses the considerations and challenges of running Apache Kafka on Kubernetes for traditional enterprises. It highlights the benefits of using Kubernetes for managing Kafka clusters, especially when other applications are already on Kubernetes, and the importance of skilled storage and network teams. It also touches upon the complexity of stateful services on Kubernetes, the need for low-latency storage, and routing messages to specific brokers. The post advocates for using tools like Helm Charts and Confluent Operator for easier deployment and management.
Aug 2018Getting Started with Apache Kafka and Kubernetes | ConfluentThis post introduces open-source Helm Charts for deploying Confluent Platform components (Kafka, ZooKeeper, Schema Registry, REST Proxy, Kafka Connect) on Kubernetes. It details the use of StatefulSets and Persistent Volumes for stateful components, and Deployments for stateless components. It also highlights best practices for running Confluent Platform on Kubernetes in a white paper, covering networking, storage, traffic, log aggregation, and metrics.
May 2018Introducing the Confluent Operator: Apache Kafka® on KubernetesIntroduces the Confluent Operator, a Kubernetes Operator that automates the deployment, management, and operations of Apache Kafka and Confluent Platform on Kubernetes. It addresses challenges of stateful applications on Kubernetes by leveraging Custom Resource Definitions and Operators, handling configuration, rolling restarts, upgrades, scaling, and data balancing.
Jan 2018Apache Mesos and Kafka Streams for Highly Scalable MicroservicesThis post details the integration of Apache Kafka and Kafka Streams with Apache Mesos (via Mesosphere's DC/OS) for building highly scalable microservice infrastructures. It explains how DC/OS can automate the deployment, management, and scaling of Kafka components and Kafka Streams applications, offering benefits like unified management, elastic scaling, and fault tolerance. It also discusses the architecture of Mesos and DC/OS, the role of frameworks like Marathon, and provides guidance on deploying Kafka clients and microservices on DC/OS, including considerations for using Kubernetes alongside Mesos.
Jul 2015Making Apache Kafka Elastic With Apache Mesos | ConfluentThis post details the development and implementation of a Kafka Mesos framework, including the Kafka Mesos Scheduler and Executor. It explains how this framework enables operational automation, scaling, and fault tolerance for Kafka clusters managed by Apache Mesos, contrasting it with the limitations of using Marathon for stateful services. It also provides installation and administration details.