Broadcom Modernizes Machine Learning & Anomaly Detection with ksqlDB

This post details a prototype integration of ksqlDB into Broadcom's Mainframe Operational Intelligence (MOI) product to improve anomaly detection and reduce false positives in metric filtering. It demonstrates how ksqlDB can be used to create metric filters based on a combination of metric instance information, values, and types. The prototype involved inserting a ksqlDB server between the Machine Learning Service and the Alarm Lifecycle Management Service, with ksqlDB filtering alert flags. The post covers the installation of ksqlDB server, creating streams with custom timestamps, selecting specific metric instances, and defining filter rules using INNER JOIN WITHIN. It also addresses challenges with Avro schema mapping, specifically casing differences and map serialization, and outlines a solution involving retrieving schemas from Confluent Schema Registry and generating corresponding POJOs.