Confluent for Kubernetes

5/11/2021

Introduces Confluent for Kubernetes, a new product that enables self-management of Confluent Platform on private infrastructure using a cloud-native, API-driven approach. It details the use of Kubernetes CustomResourceDefinitions (CRDs) to declaratively manage Confluent services (Kafka, ZooKeeper, Schema Registry, Kafka Connect, ksqlDB, Control Center) and data plane resources like topics and RBAC. The post highlights automated scaling, secure deployments, rolling upgrades, fault-tolerant infrastructure deployment, and the integration with GitOps practices. It also discusses the evolution from Helm to CRDs for a more Kubernetes-native experience.

9/24/2020

This post details how to deploy and manage Apache Kafka and Confluent Platform on Kubernetes using modern DevOps practices like GitOps and the Operator pattern. It introduces the 'streaming-ops' example project which demonstrates declarative resource management with Kubernetes Deployments and Flux for GitOps workflows. It also explains the Operator pattern and its application to managing Confluent Platform components and Kafka Connect deployments. The post highlights the benefits of declarative resource management and GitOps for managing complex streaming applications, including improved automation, auditability, and recovery from failures. It also touches upon graduated environments and Kustomize for managing different deployment stages.

9/9/2020

This post details the deployment of Confluent Platform on OpenShift Container Platform (OCP) using the Confluent Operator. It covers the prerequisites for deploying OCP on AWS, including AWS account setup, Red Hat network access, and OCP SDK installation. It then walks through the OCP cluster installation process using `openshift-install` and provides guidance on choosing persistent storage for Kafka on both on-premises and public cloud Kubernetes clusters.

5/29/2020

This post introduces three key features for elastic scaling of self-managed Kafka on Kubernetes: the Kubernetes Operator (available), Tiered Storage (preview), and Self-Balancing Clusters (coming soon). It explains how these features automate broker provisioning, storage scaling, and partition rebalancing, contrasting the manual, non-elastic approach with the automated, elastic approach. The post highlights the synergy between these features for dynamic scaling.

5/6/2020

This post introduces Project Metamorphosis, focusing on the 'Elasticity' trait. It details Confluent Cloud's move to a single usage-based billing model for all products, including Dedicated clusters, allowing dynamic provisioning and scaling. It also announces self-service expansion and automatic data balancing for Dedicated clusters. For on-prem, Confluent Platform will support automatic data balancing with Confluent Operator for Kubernetes. Contributions to Apache Kafka (KIP-500 and KIP-595) are highlighted to remove ZooKeeper dependency and enable scaling to millions of partitions.

10/4/2019

This post details the deployment and operation of Confluent Platform on Pivotal Container Service (PKS) using the Confluent Operator. It provides a step-by-step guide for setting up a Kafka cluster, including requirements, operator and Kafka setup tasks using Helm, and verification steps for both internal and external connectivity. It also briefly mentions the deployment of Schema Registry as an example of other Confluent Platform components.

10/24/2018

This post discusses the considerations and challenges of running Apache Kafka on Kubernetes, focusing on the organizational and technical aspects. It highlights the benefits of using Kubernetes for managing Kafka clusters, especially when other applications are already on Kubernetes, and emphasizes the need for cooperation from storage and network teams. It also mentions Confluent Operator as a tool to simplify deployment and management, and discusses the role of StatefulSets and the importance of persistent storage and proper networking configurations for stateful services like Kafka.

8/8/2018

This post introduces Helm Charts for deploying Confluent Platform components (Kafka, ZooKeeper, Schema Registry, REST Proxy, Kafka Connect) on Kubernetes, utilizing StatefulSets and Persistent Volumes for stateful components and Deployments for stateless ones. It also highlights a white paper on best practices for running Confluent Platform on Kubernetes, covering networking, storage, traffic, log aggregation, and metrics.

5/3/2018

Introduces the Confluent Operator, a Kubernetes Operator designed for automated deployment and management of Apache Kafka and Confluent Platform on Kubernetes. It details the challenges of running stateful applications like Kafka on Kubernetes, such as managing configuration, performing rolling restarts and upgrades, and scaling with data balancing. The Operator utilizes Kubernetes Custom Resource Definitions and the Operator API to automate these processes, ensuring stable node identity, state persistence, and application recovery. It also highlights the use of official Confluent Platform Docker images and integration with various Kubernetes distributions and cloud provider services.

1/12/2018

This post details the integration of Apache Kafka and Kafka Streams with Apache Mesos (via Mesosphere's DC/OS) for building highly scalable microservice infrastructures. It explains how DC/OS can automate the deployment, management, and scaling of Kafka components and Kafka Streams applications, offering benefits like unified management, elastic scaling, and fault tolerance. It also discusses the architecture of Mesos and DC/OS, the role of frameworks like Marathon, and provides guidance on deploying Kafka clients and microservices on DC/OS, including considerations for using Kubernetes alongside Mesos.

7/16/2015

This post details the development and implementation of a Kafka Mesos framework, including the Kafka Mesos Scheduler and Executor. It explains how this framework enables operational automation, scaling, and fault tolerance for Kafka clusters managed by Apache Mesos, contrasting it with the limitations of using Marathon for stateful services. It also provides installation and administration details.