Distributed Consensus Reloaded: Apache ZooKeeper and Replication in Apache Kafka | Confluent

What this post added

This post explains the role of distributed consensus services like Apache ZooKeeper in distributed systems, focusing on how Apache Kafka uses ZooKeeper for metadata management, replica set coordination, and leader election. It clarifies that ZooKeeper provides a 'coordination kernel' rather than an explicit consensus API, and that its atomic broadcast capabilities are fundamental to implementing replicated state machines, which Kafka uses for data replication. The post also highlights potential pitfalls in reasoning about agreement in distributed systems and provides a concrete example of how Kafka leverages ZooKeeper's sequential nodes to break ties and achieve agreement on values.