How Confluent Ensures Data Integrity for 8 Trillion Kafka Messages Per Day

This post details Confluent's approach to durability auditing for over 8 trillion Kafka messages per day in Confluent Cloud. It explains the challenges of auditing data in motion, past scenarios of data loss (replica divergence, tail data loss, metadata divergence, configuration bugs, log start offset bugs), and the lessons learned emphasizing proactive anomaly detection. The core of the post describes the "durability auditing" service, which involves real-time monitoring of sensitive operations (configuration changes, leadership changes, retention management) and weighted auditing that considers storage medium durability guarantees and metadata sensitivity. It introduces a "durability score" and details the internal process of capturing partition leader's sensitive operations, guaranteeing ordered delivery, comparing events against a durability database (source of truth), and updating the source of truth. Examples of detecting tail data loss and replica divergence due to unclean leader election are provided.