TechPrufer
Get started
Confluent
Confluent logoConfluent · Feature trail

Confluent Cloud Availability and Resilience

21 posts · Sep 2017 – Oct 2025

What this trail covers

Confluent Cloud continues to evolve its Kora Engine, a cloud-native event streaming platform designed for Kafka. This iteration focuses on enhancing reliability, elasticity, and cost-efficiency through a cellular architecture for multi-tenancy, serverless abstractions, self-balancing clusters with tiered storage, automatic mitigation for high availability, and robust data durability leveraging object storage. The platform aims to provide consistent, predictable performance across diverse cloud environments, offering a fundamentally more managed and cost-effective alternative to hosted Kafka solutions by decoupling compute and storage, enabling elastic scaling with ECKUs and diskless clusters, and optimizing networking with Private Networking Interfaces (PNI).

All Confluent trails
  1. Cost of Apache Kafka® Migrations: Estimate & Reduce Streaming Platform CostsThis post details the cost drivers of Apache Kafka migrations, emphasizing hidden costs like engineering hours, downtime risk, and dual-environment operations. It contrasts self-managed Kafka costs with Confluent Cloud, highlighting Confluent Cloud's lower TCO due to managed infrastructure, reduced engineering effort, minimal downtime via Cluster Linking, and integrated tooling. Strategies for reducing migration expenses include aggressive scope reduction (eliminating unused topics, archiving data) and planning/testing in parallel environments using Cluster Linking. The Confluent Migration Accelerator program is presented as a way to de-risk and speed up migrations, with customer examples demonstrating significant cost and time savings.
  2. Building for 99.99% Uptime on Confluent CloudThis post details Confluent's internal engineering practices for achieving 99.99% uptime on Confluent Cloud. It highlights the architectural design for multi-zone availability, including distributing Kafka topic replicas across availability zones and redundant infrastructure. The post also describes the proactive approach to resilience through continuous testing, monitoring, automated response, and failure injection (Chaos Engineering). For customers, it provides actionable advice on auditing availability requirements, integrating Confluent Cloud metrics with their observability platforms, verifying client reachability, and utilizing cluster linking for multi-cloud/multi-region replication and failover. It emphasizes transparency in platform operations and preparation for incidents.
  3. Testing & Maintaining Apache Kafka® DR and HA ReadinessThis post details the critical aspects of thoroughly testing your Kafka cluster's resilience and implementing effective strategies for long-term DR readiness. It explains the trade-offs of unclean leader election, emphasizing its risk of data loss and recommending it be kept false. It outlines how to handle network partitions using replication factor, ISRs, min.insync.replicas, and acks=all, along with network design and monitoring. The importance of time synchronization via NTP for data consistency, coordinated operations, and DR effectiveness is highlighted. Key metrics for monitoring leader election (LeaderElectionRateAndTimeMs, UncleanLeaderElectionsPerSec, OfflinePartitionsCount, UnderReplicatedPartitions, ActiveControllerCount) and DR health (cross-cluster replication lag, timestamp skew) are provided, along with management strategies like preferred replica election and addressing root causes.
  4. Cut the Costs of Hosted Kafka with Confluent Cloud’s Price GuaranteeThis post elaborates on Confluent Cloud's cost-efficiency and operational benefits, specifically highlighting how it reduces infrastructure costs by 60% or more through its cloud-native Kora Engine, elastic autoscaling from zero to massive workloads, Freight Clusters to cut cross-AZ networking fees by up to 90%, and Private Networking Interface (PNI). It also details the 'truly fully managed' aspect with instant provisioning, automated zero-downtime upgrades, per-client quotas for tenant isolation, and a comprehensive Terraform provider. The post reinforces the 99.99% uptime SLA covering the entire service, the fault-tolerant Kora Engine's decoupling of compute from storage, and the use of Cluster Linking and Schema Linking for multi-region disaster recovery. It contrasts these with the cost and operational burdens of hyperscaler-hosted Kafka services.
  5. Hosted Apache Kafka® vs. Fully Managed With Confluent CloudThis post details how Confluent Cloud's Kora engine provides a fully managed Kafka service that is fundamentally different from 'hosted' or 'provisioned' Kafka offerings. It highlights key architectural innovations that address cost-inefficient scaling, broker availability, and maintenance challenges. Specifically, it elaborates on elastic scaling with ECKUs and diskless clusters, cost-effective networking via Private Networking Interfaces (PNI), and self-healing capabilities for proactive node replacement and automatic rebalancing. The post contrasts these with the limitations of static provisioning, manual rebalancing, and cross-AZ traffic costs in hosted solutions, emphasizing Confluent Cloud's ability to deliver lower total cost of ownership and higher reliability.
  7. Use Freight Clusters and Lambda for Cost-Effective Streaming PipelinesThis post introduces Confluent Cloud Freight clusters, a new type of cluster optimized for high-throughput, relaxed latency workloads at a significantly lower cost (up to 90% cheaper than self-managed Kafka). It demonstrates how to build cost-effective, serverless streaming pipelines by combining Freight clusters for data ingestion with AWS Lambda functions for downstream processing, such as database lookups and customer notifications. The integration leverages private network interfaces (PNI) for secure connectivity and AWS Lambda's event source mapping (ESM) with Provisioned Mode for fine-tuned throughput control. The example use case involves an e-commerce survey system that captures web clicks, ingests them into Kafka, filters events using Lambda ESM, retrieves customer data from a private RDS instance, and sends survey links via SNS.
  8. Kafka Scaling Reinvented with Elastic CKUsThis post introduces Elastic Confluent Units (eCKUs) as the mechanism powering instantaneous autoscaling for Confluent Cloud's Enterprise clusters. It details how eCKUs represent a collection of capacity across throughput, partitions, and client connections, and are automatically allocated and deallocated based on real-time workload demand. The post contrasts this serverless autoscaling approach with traditional capacity management strategies (peak provisioning, predictable demand scaling) and highlights the cost savings and operational benefits of always right-sized clusters, eliminating manual intervention, over-provisioning, and downtime risk.
  9. Confluent’s Kora Engine awarded ‘Best Industry Paper’ at VLDBThis post details the Kora Engine, Confluent's next-generation platform serving the Kafka protocol for thousands of customers. It highlights Kora's cloud-native design principles, including cellular architecture for multi-tenancy and data isolation, serverless abstractions, self-balancing clusters with tiered storage for elastic expansion, automatic detection and mitigation for high availability (99.99% uptime SLA), and seamless replication/backup leveraging object storage for data durability. The paper was awarded 'Best Industry Paper' at VLDB 2023.
  10. Kora: The Cloud Native Engine for Apache KafkaThis post introduces Kora, Confluent's cloud-native engine for Apache Kafka, detailing its architecture and benefits. Kora is designed for multi-tenancy, elasticity, reliability, performance, and cost optimization, differentiating it from self-managed Kafka. Key architectural elements discussed include a multi-tenancy model with strong isolation, a stateless network routing layer for autoscaling, and intelligent data locality management across memory, SSDs, and object storage. The post highlights Kora's ability to provide a true cloud service at scale, with continuous improvements rolled out via a multi-phase fleet update software.
  11. Why Confluent? The Largest Kafka Service in the WorldThis post details Confluent's engineering capabilities in reliability, durability, scalability, and security for their Kafka service. It provides statistics on managed clusters (30,000+), daily messages (3 trillion), uptime SLA (99.99%), monitoring (7,700+ monitors), multi-zone availability, self-healing automation, Cluster Linking, and Schema Linking for disaster recovery. Durability is enhanced with checks on over 80 trillion Kafka messages per day, exceeding Apache Kafka standards through proactive design reviews, real-time auditing, and at-scale data integrity repair. Scalability is highlighted with 10x elasticity over Apache Kafka, 2,500+ cluster expansions/contractions annually, and support for up to 100 CKUs. Security features include private networking, BYOK, SSO, OAuth, RBAC (2 billion checks daily), and audit logging.
  12. Making Apache Kafka 10x More Reliable, Available, and ResilientThis post details Confluent Cloud's approach to achieving a 99.99% uptime SLA, highlighting built-in multi-zone availability by distributing Kafka topic replicas across three different availability zones. It emphasizes redundancy in underlying infrastructure, active monitoring of workload and performance metrics to manage multi-tenant clusters, and enforcement of limits to prevent noisy neighbor issues. The post also describes Confluent's investment in proactive cloud monitoring with Health Checks that emulate customer application paths, and the use of automation for self-healing, including auto-detection of cloud service unavailability and cluster rebalancing. Battle-tested operational processes are discussed, including extensive test plans, failure injection mechanisms (Trogdor+ChaosMesh), and load tests for multi-tenant environments. Finally, it introduces Cluster Linking and Schema Linking as features to architect multi-region disaster recovery plans for enhanced availability against regional outages.
  13. Elastically Scale Apache Kafka in a Single Click | Confluent CloudThis post introduces the self-serve capability to elastically scale dedicated Kafka clusters in Confluent Cloud by expanding and shrinking them in terms of CKUs. It details the user experience through the UI, CLI, and APIs, including cluster load metrics, resizing workflows, safeguards against adverse performance impacts (e.g., partition count exceeding CKU limits), billing implications, and the underlying two-step process of capacity adjustment and data rebalancing. Future plans for autoscaling based on policies are also mentioned.
  14. Cloud-Native, Multi-Tenant Apache Kafka with Confluent CloudThis post details how Confluent Cloud leverages Apache Kafka's existing multi-tenancy features (authentication, authorization, encryption, quotas, namespace isolation) to build a cloud-native multi-tenant Kafka service. It explains the requirements for cloud-native multi-tenancy, such as abstraction, pay-as-you-go models, stringent SLOs, and cost-efficiency. It describes how Confluent Cloud maps logical Kafka clusters to physical clusters, assigns tenant IDs, and enforces data, namespace, and performance isolation at the broker level through tenant metadata, ACLs, and quotas. The post also discusses capacity planning and monitoring in a multi-tenant cloud environment.
  15. Making Kafka Serverless: Lessons From Confluent CloudThis post details the architecture and implementation of Confluent Cloud's serverless Kafka offering. It describes the four key components: the Confluent Cloud control plane, the Confluent Kubernetes Operator, Self-Balancing Clusters, and Infinite Storage. The control plane manages provisioning, scaling, and API requests, translating customer intents into Kubernetes custom resources (Physical Stateful Clusters - PSCs). The Confluent Kubernetes Operator consumes PSCs to manage Kafka clusters. Self-Balancing Clusters automatically manage broker placement and storage balancing. Infinite Storage leverages tiered storage to object storage for elastic capacity. The post also discusses the microservices architecture, event-driven communication via Kafka (mothership Kafka), and the use of Kubernetes for managing services.
  16. Design Considerations for Cloud-Native Data SystemsThis post introduces the concept of cloud-native data systems and outlines key capabilities expected by users: Elasticity, Infinite scale, Resiliency, Multi-tenancy, Pay per use, Cost-effectiveness, and Global reach. It then details Confluent Cloud's approach to achieving these capabilities, specifically focusing on Elasticity through its control plane and Self Balancing Clusters (SBC) with tiered storage, and Performance/Scalability by understanding user workloads and optimizing the stack. It also elaborates on Multi-tenancy by discussing access, namespace, and performance isolation, and introduces the Storage Durability Audit as a high-level capability.
  17. Apache Kafka® Broker Failures & Other OutagesThis post details the impact of Kafka broker and cluster failures on applications, emphasizing the need for robust operational practices and application-level resilience. It outlines best practices for preventing outages (redundancy, distribution, replication, monitoring, upgrades, automation) and discusses how different application architectures (entry points vs. polling, synchronous vs. asynchronous sends) are affected by extended outages. It also describes the characteristics of extended outages and how client configurations influence error handling, providing a framework for designing applications that can gracefully handle service interruptions.
  18. Confluent Guarantees Event Streaming Reliability and Proactive SupportThis post introduces Proactive Support for Confluent Platform, which analyzes cluster metadata in real-time to identify and alert users to potential issues before they occur. It leverages the Confluent Telemetry Reporter to send metadata to Confluent for analysis, similar to how Confluent Cloud is managed. This provides Confluent support with critical context, reducing the time needed for issue resolution. The post also highlights the importance of context and experience in meeting SLAs for Kafka reliability, using hung transactions as an example of proactive monitoring and mitigation.
  19. Scaling Kafka to 10+ GB/Second in Confluent CloudThis post details Confluent Cloud's ability to scale Kafka clusters to over 10 GBps of aggregate throughput without downtime. It explains the underlying mechanisms for elastic scaling, including the concept of Confluent Units for Kafka (CKUs) which encapsulate cloud resources. The process involves provisioning new CKUs, which adds brokers and storage, followed by an automated, sophisticated partition rebalancing strategy. This strategy involves identifying candidate partitions for movement based on broker utilization, incrementing ISR counts, placing new replicas on new brokers, and then carefully managing failovers to new leaders. The post highlights the use of Kubernetes and the Confluent Operator for infrastructure automation and discusses the algorithm for rebalancing, including optimizations and the impact of leader changes on client connections.
  20. Elastic Kafka Clusters in Confluent Cloud - Project MetamorphosisThis post announces Project Metamorphosis, focusing on the 'Elasticity' trait of cloud-native data systems. It introduces usage-based billing for all Confluent Cloud products, allowing dynamic provisioning and de-provisioning of Dedicated clusters. It also announces self-service expansion and automatic data balancing for Dedicated Kafka clusters in Confluent Cloud. For on-prem, Confluent Platform will support automatic data balancing with Confluent Operator for Kubernetes-like elasticity. Contributions to Apache Kafka, specifically KIP-500 and KIP-595 for a Raft-based metadata quorum, are highlighted as foundational improvements for elasticity and scalability.
  21. Stable, Secure Kafka as a Service: Confluent CloudThis post details the rigorous testing and release processes employed by Confluent to ensure the stability and security of Confluent Cloud. It covers community contributions to Apache Kafka (KIPs, committer reviews, time-based releases), Confluent's internal testing suites (unit, integration, system, cloud system, performance, and soak tests), and the phased road to production with canary releases. The emphasis is on the comprehensive quality assurance measures taken to deliver a reliable managed Kafka service.
  22. Building Confluent Cloud – Here’s What We LearnedThis post details the current architecture of Confluent Cloud, focusing on the data plane network architecture with Layer 4 load balancers for broker abstraction and the control plane's mothership/satellite components managing physical clusters via Kubernetes custom resources. It also outlines improvements made to Apache Kafka based on Confluent Cloud learnings, including TLS certificate renewals without broker restarts, multi-tenant Kafka, and client DNS re-resolution.
  23. Why Scrapinghub’s AutoExtract Chose Confluent CloudThis post details Scrapinghub's migration of their AutoExtract service to Confluent Cloud. It explains their use case involving web scraping, rendering, and AI-driven data extraction, highlighting how Kafka is used to partition and distribute workloads for downloader and AI extraction engine components. The decision to use Confluent Cloud over self-hosting Kafka on Kubernetes or Amazon MSK is justified by factors like offloading infrastructure management, consumption-based pricing, vendor independence, and access to Confluent-specific features. The post also touches on initial setup, load testing, and workarounds for missing tools like Kafka Manager and Burrow, including the use of a consumer metrics exporter.
  24. Free Apache Kafka as a Service with Confluent CloudThis post introduces Confluent Cloud as a serverless offering for Apache Kafka, available on GCP, Azure, and AWS. It highlights three key aspects of the serverless experience: focusing on outcomes rather than managing clusters, elastic scaling that grows with load, and consumption-based pricing (per GB data in, data out, and data retained). It also mentions the 99.95% SLA and different support plan tiers.
  25. The Rise of Managed Apache Kafka as a ServiceThis post defines and differentiates between fully managed, partially managed, and hosted Kafka services. It highlights the complexities of self-managing Kafka in the cloud (compute instance selection, storage sizing, security, availability zones, recoverability, observability, data regulations, developer onboarding) and positions Confluent Cloud as a fully managed service that abstracts these complexities. Key characteristics of a fully managed service are detailed, including minimal user involvement in infrastructure decisions, simplified onboarding (e.g., 90 seconds to start), and the absence of co-responsibility for operational tasks like patching, upgrades, or managing ecosystem components like Schema Registry and Kafka Connect.
  26. Confluent Cloud: Enterprise-Ready, Hosted Apache Kafka is Here!This post announces the general availability of Confluent Cloud on AWS, a managed Kafka service. It highlights a 99.95% uptime guarantee and describes how Confluent Cloud provides private Kafka clusters, handling deployment, monitoring, and upgrades. Users configure throughput, availability zones, and data retention. The post also mentions the Confluent Cloud CLI for administrative tasks and directs readers to the product page or a contact form for setup.
  27. Apache Kafka® Disaster Recovery | Confluent Best PracticesThis post details Apache Kafka disaster recovery best practices, focusing on multi-region deployments. It introduces Confluent Cloud features like Cluster Linking and Schema Linking for building multi-region, multi-cloud, and hybrid cloud deployments. It also highlights Confluent Replicator for managing cross-datacenter replication, timestamp preservation, preventing cyclic topic repetition, resetting consumer offsets, and centralized schema management. Architecture designs for multi-datacenter disaster recovery, including active-passive and active-active patterns, are discussed.