Ensuring data reliability and observability in risk systems

This post details the implementation of Apache Flink SQL for real-time data aggregation and transformation of heterogeneous JSON data from various upstream services feeding into the GrabDefence risk management platform. It describes the use of custom table functions like JSONEXPLOAD to deconstruct nested JSON, 5-minute tumbling windows for aggregation, and the integration of Flink's output with Datadog for anomaly detection and alerting via Slack. The post also covers strategies for organizing Datadog monitors by service stream and leveraging its Anomaly Detection feature to quickly identify data quality issues.