Exactly-once Semantics is Possible: Here's How Apache Kafka Does it

This post introduces and explains exactly-once semantics (EOS) in Apache Kafka, detailing how the 0.11 release and Confluent Platform 3.3 enable this capability. It elaborates on the challenges of achieving EOS in distributed systems and breaks down Kafka's solution into two key components: 1. Idempotent Producers: This feature ensures that retried sends of the same message batch only result in a single write to the Kafka log per partition, preventing duplicates due to producer or broker errors. It works by persisting sequence numbers to the replicated log. 2. Transactions API: This allows for atomic writes across multiple partitions, ensuring that either all messages in a batch are visible or none are. It also enables committing consumer offsets within the same transaction for end-to-end EOS. The post also explains how the Kafka Streams API leverages these features with the `processing.guarantee=exactly_once` configuration for exactly-once stream processing, guaranteeing that each received record's processed results are reflected once, even under failures.