Improving Stream Data Quality with Protobuf Schema Validation | Confluent

What this post added

This post details Deliveroo's 'Franz' framework, which enforces Protobuf schema validation and evolution for inter-service communication over Kafka. It describes the need for structured message formats, the evaluation of serialization formats (Thrift, Protobuf, Avro), and the decision to use Protobuf. The post elaborates on Protobuf's forwards and backwards compatibility features and outlines specific schema evolution rules enforced through unit tests: immutability of field numbers, no type amendments, reserving removed fields, and not amending field names. It also introduces the use of Protobuf custom options to define topic ownership and message formats, inspired by Confluent Schema Registry.