Kafka Needs No Keeper - Removing ZooKeeper Dependency

What this post added

This post details the technical plan and roadmap for removing ZooKeeper dependency from Apache Kafka through KIP-500. It explains the problems with external metadata management (duplication, inefficiency, scalability limits, de-synchronization) and proposes storing metadata in a Kafka partition managed by a Raft quorum. It outlines the controller architecture changes, including standby controllers and O(1) topic operations. The roadmap includes removing ZooKeeper from administrative tools, implementing self-managed metadata quorums with a pull-based Raft protocol (KIP-595), introducing KIP-500 mode (initially experimental), and a bridge release strategy for zero-downtime upgrades using new controller APIs like AlterIsr.