Broadcom Modernizes Machine Learning & Anomaly Detection with ksqlDB

What this post added

This post details how Broadcom modernized its Machine Learning & Anomaly Detection system (Mainframe Operational Intelligence - MOI) by integrating ksqlDB. It describes the architecture of MOI, which uses Kafka for inter-service communication and Schema Registry for schema management. The post outlines a prototype where ksqlDB is inserted between the Machine Learning Service and the Alarm Lifecycle Management Service to filter alert flags based on metric values and instance information. It covers the installation of ksqlDB server, creating streams with custom timestamps, selecting specific metric instances, and defining filter rules using INNER JOIN WITHIN for time-bound correlation. A key technical challenge addressed is the schema transformation required for ksqlDB to output data compatible with downstream services, including handling casing differences and Avro map serialization, which was resolved by retrieving and regenerating POJOs from Schema Registry.