How Confluent Ensures Data Integrity for 8 Trillion Kafka Messages Per Day

What this post added

This post details Confluent's durability auditing mechanisms for proactively detecting data integrity issues on trillions of Kafka messages per day. It covers real-time monitoring of sensitive operations, weighted auditing based on storage mediums, and a durability score based on message durability. It also provides examples of tail data loss and replica divergence due to unclean leader elections and how these are detected.