Real-time data quality monitoring: Kafka stream contracts with syntactic and semantic test

Introduced a real-time data quality monitoring solution for Kafka streams. This system allows users to define data contracts including schema agreements, semantic rules (string patterns, number ranges, constant values), and ownership details. It uses FlinkSQL to execute tests based on these contracts, identifying syntactic (schema mismatches) and semantic (field-level rule violations) issues in real-time. Problematic data is published to a dedicated Kafka topic and sunk to S3 for observability. Alerts are sent to stream owners via Slack through the Genchi platform. LLM-based recommendations are provided to simplify semantic rule definition. The solution has been deployed across 100+ Kafka topics.