The Curious Incident of the State Store in Recovery in ksqlDB

What this post added

This post details a bug in ksqlDB's state store recovery mechanism where an optimization for avoiding changelog topics led to incorrect schema registration with Schema Registry during node relocation. The bug (KAFKA-10179) caused deserialization errors when ksqlDB attempted to read from the state store using the wrong schema ID and subject. A temporary fix involves registering the schema ID under the phantom changelog subject. The future roadmap includes improving ksqlDB's integration with Schema Registry to handle custom internal schemas more robustly.