Infinite Kafka Data Retention in Confluent Platform

What this post added

This post introduces Tiered Storage in Confluent Platform 5.4, a feature that separates data storage from data processing by offloading data to remote object stores like Amazon S3. This improves scalability and elasticity by reducing the data that needs to be replicated during rebalancing operations. It enables Kafka to serve as a system of record with infinite retention, supporting use cases like materialized views, ML training, and regulatory compliance. The implementation ensures durability, integrity, seamless operation for users, portability across storage backends, and performance by managing data tiering and retrieval. A 'hotset' concept is introduced to keep recent data on local disks for low-latency access, while older data is archived. The leader of a partition manages the tiering process, with metadata stored in the `_confluent-tier-state` topic.