TechPrufer
Get started
Confluent
Confluent logoConfluent · Feature trail

Kafka Infinite Storage

24 posts · Dec 2015 – Jan 2024

What this trail covers

Confluent Platform introduces Tiered Storage, enabling operators to offload the majority of Kafka data to remote object stores like Amazon S3. This separation of storage and processing enhances scalability, elasticity, and cost-efficiency by reducing the burden of rebalancing, cluster expansion/shrinking, and broker replacement. Tiered Storage allows for infinite data retention, enabling Kafka to serve as a system of record for use cases like materialized views, ML model training, and regulatory compliance. This post details the initial considerations for deploying Kafka on AWS EBS, highlighting the importance of storage performance and reliability for Kafka clusters. It specifically discusses the benefits of EBS volumes for sequential I/O workloads and the cost-effectiveness of st1 and sc1 offerings.

All Confluent trails
  1. Apache Kafka® vs Confluent Cloud: Latency Benchmarking ResultsThis post benchmarks Confluent Cloud's Kora engine against Apache Kafka 3.6 with KRaft, demonstrating up to 16x improvement in p99 E2E latencies across various workload profiles and throughput scales. It details the benchmarking methodology, including the use of health check services for latency measurement and the setup for comparing Confluent Cloud's CKU-based offering with similarly resourced Apache Kafka clusters. The benchmarks show Confluent Cloud's performance consistency and predictability under varying throughput and partition counts, highlighting its advantages over Apache Kafka in steady-state performance.
  2. Lowering Kafka Costs by Improving Utilization and Ease of UseThis post details how Confluent Cloud reduces the cost to run data streaming workloads by achieving efficient resource utilization (up to 3x compared to self-managing on your own), while reducing the complexity for customers to manage and scale their clusters. It details how multi-tenancy, serverless abstractions, and elasticity are leveraged to improve overall service utilization and lower the infrastructure costs of running Kafka. Confluent Cloud uses serverless abstractions across compute, storage, and networking to continually improve system utilization while saving customers time and complexity. Elasticity is achieved by decoupling compute and storage layers, enabling dynamic scaling and continuous rebalancing of partitions across clusters.
  3. Kafka Storage That’s 10x More Scalable and Elastic with ConfluentThis post introduces and details Confluent Cloud's 'Infinite Storage' engine, which separates compute and storage by offloading data to cloud object storage. It addresses operational pain points of traditional Kafka storage, such as throttling, capacity balancing, and retention negotiation. The post highlights the benefits of this architecture, including automatic storage growth, cost savings through pay-as-you-go, and enabling new use cases like using Kafka as a source of truth and powering real-time/historical analytics. It also explains performance improvements achieved through resource isolation between real-time and historical consumers.
  4. Making Confluent Cloud 10x More Elastic Than Apache Kafka | Confluent Making Confluent Cloud 10x More Elastic Than Apache KafkaThis post details how Confluent Cloud achieves 10x greater elasticity than Apache Kafka through its Intelligent Storage system. It explains that by keeping most data in object storage and a smaller fraction on local disks, Confluent Cloud significantly reduces the amount of data that needs to be moved during cluster scaling operations. This leads to scaling times that are up to 30x faster than vanilla Apache Kafka. The post also highlights that this elasticity applies to both scaling up and down, improving resiliency by enabling faster recovery from failures and quicker fleet-wide upgrades. Real-world data shows that Confluent Intelligent Storage decreased cluster expansion times by an average of 69%, with extreme cases seeing a 954% decrease.
  5. How Instacart Used Confluent to Scale 10 Years of Growth in 6 WeeksInstacart details their experience scaling their event streaming infrastructure with Confluent Cloud to handle a massive surge in demand. They migrated client streams to Kafka, focusing on topic design, infrastructure resilience, and partitioning. The use of Confluent Cloud's serverless architecture and Infinite Storage allowed for rapid scaling and simplified storage management. This enabled real-time data integration for inventory management, predictive models, and fraud detection, supporting their move towards a data mesh architecture.
  7. How Confluent Ensures Data Integrity for 8 Trillion Kafka Messages Per DayThis post details Confluent's approach to durability auditing for over 8 trillion Kafka messages per day in Confluent Cloud. It explains the challenges of auditing data in motion, past scenarios of data loss (replica divergence, tail data loss, metadata divergence, configuration bugs, log start offset bugs), and the lessons learned emphasizing proactive anomaly detection. The core of the post describes the "durability auditing" service, which involves real-time monitoring of sensitive operations (configuration changes, leadership changes, retention management) and weighted auditing that considers storage medium durability guarantees and metadata sensitivity. It introduces a "durability score" and details the internal process of capturing partition leader's sensitive operations, guaranteeing ordered delivery, comparing events against a durability database (source of truth), and updating the source of truth. Examples of detecting tail data loss and replica divergence due to unclean leader election are provided.
  8. From Apache Kafka to Confluent Cloud: Optimizing for Speed, Scale, and StorageThis post details Confluent's experience optimizing Apache Kafka for Confluent Cloud, covering guiding principles for incremental optimizations and architectural improvements for step-changes in performance. It discusses workload simulation using Trogdor, performance envelopes based on message characteristics, topic-partition distribution, cluster topology, number of connections, and utilization, as well as nightly performance tests. It also covers infrastructure considerations including hardware selection, OS tuning (io_uring, dirty_background_bytes, dirty_bytes), and JVM tuning.
  9. Making Kafka Serverless: Lessons From Confluent CloudThis post introduces the architectural components and design considerations for making Kafka serverless in Confluent Cloud. It details the roles of the control plane, Confluent Kubernetes Operator, Self-Balancing Clusters, and Infinite Storage. The control plane manages resource provisioning and scaling by translating customer requests into Kubernetes custom resources (Physical Stateful Clusters). Self-Balancing Clusters automate elastic expansion/contraction and storage balancing. Infinite Storage leverages tiered storage with object storage for elasticity and cost reduction. The communication between control and data planes uses an event-driven architecture with Kafka and Debezium connectors.
  10. Property Based Testing Confluent Cloud Storage for Fun and SafetyThis post details the application of property-based testing, specifically ScalaCheck's stateful testing functionality, to verify the correctness of Confluent Cloud's Tiered Storage feature. It outlines the setup of initial state generators, action generators (Append, AdvanceHighWatermark, RollSegment, TierUploadSegment, CommitTierState, IncrementLogStartOffset, Retention, TierRetention, CloseAndReopen, Read), and the specific properties checked to ensure data integrity, prevent premature deletion, and guarantee correct data exposure to followers and consumers. The focus is on testing the log-orientated storage layer and its integration with Kafka's traditional storage.
  11. Cloud-Like Flexibility, Performance, and Scale On-Prem with Confluent and Pure StorageThis post announces the general availability of Tiered Storage for Confluent Platform, with initial support for Amazon S3 and Google Cloud Storage, and now adds support for on-premises object storage via Pure Storage FlashBlade. Tiered Storage separates compute from storage by tiering data to remote object storage, enabling elastic scaling of Kafka brokers and infinite data retention. Performance tests with Pure FlashBlade demonstrated up to 7.5 GB/sec sustained throughput and validated data correctness under high load. The integration allows for cloud-like elasticity on-premises, enabling faster analytics and historical queries.
  12. Infinite Storage in Kafka: Unlimited Retention in Confluent CloudThis post announces the rollout of unlimited storage for Confluent Cloud Kafka clusters, a feature that decouples storage from compute resources. Users will now pay only for the data produced, with no additional charges or provisioning units for storage. This enables longer data retention periods without impacting infrastructure costs or performance isolation, allowing Kafka to act as a system of record for both real-time and historical data access.
  13. Unlock Legacy Data with Mainframe Modernization Using Apache KafkaThis post introduces the concept of using Apache Kafka for mainframe offload, particularly in the context of modernizing legacy systems to handle increased demand. It highlights Kafka's ability to act as a buffer, persist and replay data, and absorb spikes in demand. The post also mentions Confluent Cloud and Confluent Platform as managed solutions for easier deployment and management, and Control Center for monitoring. It touches upon RBAC for security and ksqlDB for midstream data enrichment.
  14. Streaming Machine Learning with Tiered StorageThis post introduces the concept of streaming machine learning, where data is consumed directly from Kafka into ML frameworks like TensorFlow, eliminating the need for a separate data lake. It highlights the benefits of this approach for scalability, reliability, and reduced operational effort. It also discusses how Tiered Storage complements this by providing cost-effective long-term storage for Kafka data, enabling reprocessing for various use cases and simplifying ML infrastructure.
  15. Infinite Kafka Data Retention in Confluent PlatformThis post introduces Tiered Storage in Confluent Platform 5.4, a feature that separates Kafka data storage from processing. It explains the motivation behind this feature, focusing on improved scalability, elasticity, and cost-efficiency by offloading data to remote stores like S3. The post details how Tiered Storage enables Kafka to act as a system of record with infinite retention, facilitating use cases like materialized views and ML model training. It also discusses easier infrastructure planning and outlines the goals for implementation: durability, integrity, seamless operation for operators and developers, portability, and performance. The implementation details include tiering data using segment files and a 'hotset' for local disk caching, fetching data from remote storage, and garbage collection.
  16. Handling GDPR with Apache Kafka: How does a log forget? | ConfluentIntroduces the use of Kafka's compacted topics as a mechanism for implementing GDPR's 'right to be forgotten'. Explains that messages can be marked for deletion by sending a new message with the same key and a null value. Discusses strategies for mapping data to keys, handling composite keys, and an alternative approach using per-user encryption keys. Also covers propagating deletes from originating databases via Kafka Connectors and provides tuning configurations for the log cleaner to control compaction timing and I/O.
  17. It's Okay To Store Data In KafkaThis post argues that Kafka is designed for long-term data storage, not just messaging. It highlights use cases like event sourcing, materialized views for stream processing, and Change Data Capture (CDC) where Kafka's log compaction and persistent storage are beneficial. The author contrasts Kafka's architecture (distributed filesystem/database-like) with traditional message queues, emphasizing its ability to store, replicate, and serve data reliably. It also touches on the operational rigor required when treating Kafka as a primary store and mentions Confluent's support and hosted services.
  18. Log Compaction | Highlights in the Apache Kafka and Stream Processing Community | November 2016 | ConfluentThis post discusses KIP-87, which proposes adding a 'Compaction Tombstone Flag' to Kafka's log compaction mechanism. Currently, deleting a key from a compacted topic requires sending a record with a null value, which has limitations. The proposed flag would explicitly mark a message for deletion, improving the handling of null values and messages with 'magic bytes' or versions. The post also mentions KIP-72, KIP-81, KIP-84, and KIP-85, which relate to memory management, authentication, and client configuration.
  19. Log Compaction | Highlights in the Apache Kafka and Stream Processing Community | October 2016 | ConfluentThis post highlights several key Kafka Improvement Proposals (KIPs) that are being discussed and merged into upcoming releases, specifically focusing on Apache Kafka 0.10.1.0. These include KIP-74 (Fetch Response Size Limit), KIP-79 (Timestamp Search for New Consumer), KIP-82 (Record Headers), KIP-83 (Multiple SASL Clients in JVM), and KIP-85 (Dynamic JAAS Configuration). It also references external content on Kafka usage in GCP, stream processing architectures, and testing Kafka Streams.
  20. Use Apache Kafka to Migrate to AWS Cloud | ConfluentThis post details how Apache Kafka can be used to facilitate gradual cloud migrations by acting as a central pipeline for synchronizing data between on-premise datacenters and AWS. It highlights the use of MirrorMaker for replicating data to a Kafka cluster in AWS, and Kafka Connect for integrating with various data stores. The benefits of this architecture include proven long-term multi-datacenter support, continuous low-latency synchronization, centralized manageability and monitoring, enhanced security and governance, and cost savings. The post emphasizes that cloud migrations are often a multi-year process and advocates for a holistic, central-pipeline approach using Kafka.
  21. Deploying Apache Kafka on AWS Elastic Block Store (EBS) | ConfluentThis post discusses the critical role of underlying infrastructure, specifically storage, for Apache Kafka deployments. It highlights the importance of consistent I/O performance and reliability for Kafka brokers, which are optimized for sequential operations. The post recommends AWS EBS volumes, particularly the st1 and sc1 offerings, for their cost-effectiveness and suitability for sequential I/O workloads, enabling easier re-assignment of volumes to new instances in case of failure and preventing data loss during clean shutdowns.
  22. Log Compaction | Highlights in the Kafka and Stream Processing Community | January 2016 | ConfluentThis post details the release of Kafka 0.9 and Confluent Platform 2.0. It highlights KIP-40, which adds protocol support for managing consumer groups, enabling clients to list groups and their members/lag without direct ZooKeeper interaction. It also discusses KIP-32, which aims to add a timestamp field to Kafka messages for physical time tracking. The post also introduces Kafka Connect, a new feature in 0.9 for integrating Kafka with external data systems, and provides a tutorial on building an ETL pipeline using Kafka Connect. It mentions a benchmark comparing stream processing frameworks and Microsoft's use of Kafka in Bing. The post also announces the upcoming Kafka Summit and the availability of 'Kafka, The Definitive Guide'.