ksqlDB Execution Plans: Move Fast But Don’t Break Things

What this post added

This post introduces the concept of ksqlDB execution plans as a mechanism to manage query compatibility during upgrades and feature rollouts. It details how ksqlDB compiles queries into an internal, JSON-serializable execution plan (a DAG of steps) instead of directly building a Kafka Streams topology. This allows for independent evolution of the ksqlDB parsing and planning layer from the Kafka Streams topology generation layer, ensuring that running queries are not broken by upgrades. The post explains the motivation for this approach, the limitations of previous compatibility solutions, and the testing strategies (schema verification and QTTs) employed to maintain the integrity of the execution plan API.