Infinite Kafka Data Retention in Confluent Platform

This post introduces Tiered Storage in Confluent Platform 5.4, a feature that separates Kafka data storage from processing. It explains the motivation behind this feature, focusing on improved scalability, elasticity, and cost-efficiency by offloading data to remote stores like S3. The post details how Tiered Storage enables Kafka to act as a system of record with infinite retention, facilitating use cases like materialized views and ML model training. It also discusses easier infrastructure planning and outlines the goals for implementation: durability, integrity, seamless operation for operators and developers, portability, and performance. The implementation details include tiering data using segment files and a 'hotset' for local disk caching, fetching data from remote storage, and garbage collection.