Kafka Infinite Storage

5/9/2023

This post details how Confluent Cloud reduces the cost to run data streaming workloads by achieving efficient resource utilization (up to 3x compared to self-managing on your own), while reducing the complexity for customers to manage and scale their clusters. It details how multi-tenancy, serverless abstractions, and elasticity are leveraged to improve overall service utilization and lower the infrastructure costs of running Kafka. Confluent Cloud uses serverless abstractions across compute, storage, and networking to continually improve system utilization while saving customers time and complexity. Elasticity is achieved by decoupling compute and storage layers, enabling dynamic scaling and continuous rebalancing of partitions across clusters.

7/13/2022

Introduces and details the 'Infinite Storage' architecture for Confluent Cloud, which separates compute and storage by leveraging cloud object storage. This enables unlimited data retention, eliminates operational concerns around storage limits and throttling, and improves performance by isolating real-time and historical data access. The post also discusses customer adoption and new use cases enabled by this capability.

5/31/2022

This post introduces Confluent Cloud Intelligent Storage as the core technology enabling 10x elasticity compared to Apache Kafka. It details how Intelligent Storage utilizes tiered cloud storage (object storage and local disks) to minimize data rebalancing during scaling operations. The post quantifies the performance improvements, showing up to 30x faster scaling than vanilla Kafka and an average reduction of 69% in cluster expansion times with Intelligent Storage enabled. It also highlights how this elasticity improves resiliency by enabling faster recovery from broker failures and more efficient fleet-wide upgrades. The post contrasts the scaling process of Apache Kafka with Confluent Cloud, emphasizing the reduction in operational burden and capacity planning.

1/20/2022

Instacart details their experience scaling their event streaming infrastructure with Confluent Cloud to handle a massive surge in demand. They migrated client streams to Kafka, focusing on topic design, infrastructure resilience, and partitioning. The use of Confluent Cloud's serverless architecture and Infinite Storage allowed for rapid scaling and simplified storage management. This enabled real-time data integration for inventory management, predictive models, and fraud detection, supporting their move towards a data mesh architecture.

7/30/2021

This post details Confluent's durability auditing mechanisms for proactively detecting data integrity issues on trillions of Kafka messages per day. It covers real-time monitoring of sensitive operations, weighted auditing based on storage mediums, and a durability score based on message durability. It also provides examples of tail data loss and replica divergence due to unclean leader elections and how these are detected.

7/28/2021

This post details Confluent's experience optimizing Apache Kafka for Confluent Cloud, covering guiding principles for incremental optimizations and architectural improvements for step-changes in performance. It discusses workload simulation using Trogdor, performance envelopes based on message characteristics, topic-partition distribution, cluster topology, number of connections, and utilization, as well as nightly performance tests. It also covers infrastructure considerations including hardware selection, OS tuning (io_uring, dirty_background_bytes, dirty_bytes), and JVM tuning.

7/27/2021

This post introduces the architectural components and design considerations for making Kafka serverless in Confluent Cloud. It details the roles of the control plane, Confluent Kubernetes Operator, Self-Balancing Clusters, and Infinite Storage. The control plane manages resource provisioning and scaling by translating customer requests into Kubernetes custom resources (Physical Stateful Clusters). Self-Balancing Clusters automate elastic expansion/contraction and storage balancing. Infinite Storage leverages tiered storage with object storage for elasticity and cost reduction. The communication between control and data planes uses an event-driven architecture with Kafka and Debezium connectors.

1/12/2021

This post introduces the application of property-based testing, specifically stateful testing with ScalaCheck, to validate the safety and correctness of Confluent Cloud's Tiered Storage feature. It details the design of an initial state generator, action generators (including Append, Retention, TierUploadSegment, CommitTierState), and the core properties checked to ensure data integrity, prevent premature deletion, and verify correct data exposure. The post also explains the concepts of log segments, retention, and the high watermark in the context of this testing methodology.

10/14/2020

This post announces and details the integration of Confluent Tiered Storage with Pure Storage's FlashBlade for on-premises object storage. It covers the performance testing (7.5 GB/sec sustained throughput) and reliability testing (data correctness under high load) conducted to certify FlashBlade. The post also explains how this integration enables cloud-like elasticity and infinite retention for Kafka on-premises, facilitating use cases like analytics and hydrating downstream systems.

7/1/2020

This post introduces the concept of 'infinite storage' for Apache Kafka in Confluent Cloud, decoupling storage from compute and allowing for unlimited data retention without additional charges or pre-provisioning. It explains the technical benefits for use cases like system of record, regulatory compliance, and rebuilding ksqlDB state. The underlying technology is based on Tiered Storage, with performance optimizations in Confluent Cloud to ensure cost-effectiveness, ease of use, and performance isolation between real-time and historical data consumers.

6/16/2020

This post details how Apache Kafka can be used for mainframe offload and modernization, particularly in scenarios with high demand spikes like state unemployment systems. It explains Kafka's role as a buffer and playback mechanism. It also highlights Confluent Cloud and Confluent Platform as solutions for managing Kafka infrastructure, including Confluent Operator for Kubernetes and Control Center for monitoring. The post also mentions Role-Based Access Control (RBAC) for securing sensitive mainframe data and ksqlDB for real-time data enrichment.

1/28/2020

This post introduces the concept of streaming machine learning, where data is consumed directly from Kafka into ML frameworks like TensorFlow, eliminating the need for a separate data lake. It highlights the benefits of this approach for scalability, reliability, and reduced operational effort. It also discusses how Tiered Storage complements this by providing cost-effective long-term storage for Kafka data, enabling reprocessing for various use cases and simplifying ML infrastructure.

1/23/2020

This post introduces Tiered Storage in Confluent Platform 5.4, a feature that separates data storage from data processing by offloading data to remote object stores like Amazon S3. This improves scalability and elasticity by reducing the data that needs to be replicated during rebalancing operations. It enables Kafka to serve as a system of record with infinite retention, supporting use cases like materialized views, ML training, and regulatory compliance. The implementation ensures durability, integrity, seamless operation for users, portability across storage backends, and performance by managing data tiering and retrieval. A 'hotset' concept is introduced to keep recent data on local disks for low-latency access, while older data is archived. The leader of a partition manages the tiering process, with metadata stored in the `_confluent-tier-state` topic.

12/8/2017

Introduces the use of Kafka's compacted topics as a mechanism for implementing GDPR's 'right to be forgotten'. Explains that messages can be marked for deletion by sending a new message with the same key and a null value. Discusses strategies for mapping data to keys, handling composite keys, and an alternative approach using per-user encryption keys. Also covers propagating deletes from originating databases via Kafka Connectors and provides tuning configurations for the log cleaner to control compaction timing and I/O.

9/15/2017

This post argues that Kafka is suitable for long-term data storage, not just as a messaging queue. It explains use cases like event sourcing, populating caches, recomputing stream processing results, and Change Data Capture (CDC). The post highlights Kafka's design for persistence, disk storage, checksumming, and replication for fault tolerance, noting that performance does not degrade with data accumulation. It contrasts Kafka with traditional messaging systems that lack robust storage and replication, and positions Kafka as a 'Streaming Platform' rather than a message queue. The post also discusses Kafka's role as a commit log in a datacenter's 'giant database' architecture, with derived indexes and views served by other systems, and mentions Kafka Streams and KSQL for materialized views and queries.

5/23/2017

This post introduces the concept of using Kafka's compacted topics to implement GDPR's 'right to be forgotten'. It explains how messages can be marked for deletion via their keys and subsequently removed during the compaction process. Alternative strategies like encrypting data with user-specific keys and deleting those keys, as well as handling deletes in downstream databases via Kafka Connectors, are also discussed. Tuning parameters for compaction are provided.

3/9/2017

This post highlights KIP-107, which adds a `purgeDataBefore()` API in AdminClient, allowing developers to request data purging from Kafka beyond time-based and size-based cleanup policies. This is useful for multi-step stream processing jobs to remove intermediate data after it has been processed by downstream jobs.

11/3/2016

This post discusses KIP-87, which proposes adding a 'Compaction Tombstone Flag' to Kafka's log compaction mechanism. Currently, deleting a key from a compacted topic requires sending a record with a null value, which has limitations. The proposed flag would explicitly mark a message for deletion, improving the handling of null values and messages with 'magic bytes' or versions. The post also mentions KIP-72, KIP-81, KIP-84, and KIP-85, which relate to memory management, authentication, and client configuration.

10/11/2016

This post highlights several key Kafka Improvement Proposals (KIPs) that are being discussed and merged into upcoming releases, specifically focusing on Apache Kafka 0.10.1.0. These include KIP-74 (Fetch Response Size Limit), KIP-79 (Timestamp Search for New Consumer), KIP-82 (Record Headers), KIP-83 (Multiple SASL Clients in JVM), and KIP-85 (Dynamic JAAS Configuration). It also references external content on Kafka usage in GCP, stream processing architectures, and testing Kafka Streams.

9/22/2016

This post details how Apache Kafka can be used to facilitate gradual cloud migrations by acting as a central pipeline for synchronizing data between on-premise datacenters and AWS. It highlights the use of MirrorMaker for replicating data to a Kafka cluster in AWS, and Kafka Connect for integrating with various data stores. The benefits of this architecture include proven long-term multi-datacenter support, continuous low-latency synchronization, centralized manageability and monitoring, enhanced security and governance, and cost savings. The post emphasizes that cloud migrations are often a multi-year process and advocates for a holistic, central-pipeline approach using Kafka.

4/19/2016

This post details the performance and reliability considerations for deploying Kafka on AWS Elastic Block Store (EBS), specifically highlighting the benefits of st1 and sc1 EBS offerings for sequential I/O workloads and the operational advantages over EC2 Local Instance Storage for re-replication and clean shutdowns.

4/1/2016

This post details the merging of KIP-36 (rack-aware replica assignment), KIP-51 (Connect REST API for listing connectors), KIP-45 (standardizing container types in the new consumer API), and KIP-43 (SASL enhancements). It also highlights active discussions on KIP-4 (administrative operations API), KIP-35 (retrieving protocol version), and KIP-33 (time-based log index for purging and lookup). The technical preview of Kafka Streams is also announced.

1/1/2016

This post details the release of Kafka 0.9 and Confluent Platform 2.0. It highlights KIP-40, which adds protocol support for managing consumer groups, enabling clients to list groups and their members/lag without direct ZooKeeper interaction. It also discusses KIP-32, which aims to add a timestamp field to Kafka messages for physical time tracking. The post also introduces Kafka Connect, a new feature in 0.9 for integrating Kafka with external data systems, and provides a tutorial on building an ETL pipeline using Kafka Connect. It mentions a benchmark comparing stream processing frameworks and Microsoft's use of Kafka in Bing. The post also announces the upcoming Kafka Summit and the availability of 'Kafka, The Definitive Guide'.

10/12/2015

This post details community discussions and upcoming features for Apache Kafka 0.10.1.0, including KIPs related to fetch response size limits, timestamp searching in the new consumer, adding record headers, supporting multiple SASL authenticated Java clients in a single JVM, and dynamic JAAS configuration for Kafka clients. It also mentions Google's use of Kafka in GCP, Strata announcements, O'Reilly's insights on streams architecture, a customer 360 architecture tutorial, testing Kafka. Specifically, it highlights KIP-31 (Move to relative offsets in compressed message sets), KIP-32 (Add CreateTime and LogAppendTime to Kafka messages), and KIP-33 (Add time-based log index), which together enable time-based message lookup. It also notes that authorization patches for topic-level authorization have been merged into trunk for Kafka 0.9.0.0.