Consistent Metastore Recovery for ksqlDB Using Kafka Transactions

What this post added

This post details the implementation of Kafka transactions to ensure consistent metastore recovery for ksqlDB in interactive mode. It addresses the previous design's flaws where the command topic acted as both a request queue and a log, leading to statement validation and execution errors. The new protocol treats the command topic strictly as a log by performing validation before enqueueing and isolating servers using Kafka's transactional.id. This ensures that only one server can validate and produce statements to the command topic at a time, preventing race conditions and guaranteeing deterministic metastore recreation.