Distributed Consensus Reloaded: Apache ZooKeeper and Replication in Apache Kafka | Confluent

This post explains the fundamental problem of distributed consensus and how systems like ZooKeeper solve it by implementing atomic broadcast. It details how Kafka leverages ZooKeeper for metadata management, including broker sets, leader election, and eligible leader subsets. The post also highlights potential pitfalls in reasoning about consensus and agreement, using an example of workers consuming different flavors due to timing differences in ZooKeeper reads. It contrasts ZooKeeper's role as a 'coordination kernel' with explicit consensus APIs and explains how Kafka uses ZooKeeper's sequential nodes and state updates to achieve agreement on leader replicas.