Real-Time Risk Management Using Apache Kafka and Event Streaming

This post details the migration of Tesco PLC's Risk Engine from a monolithic, in-memory JMS-based system to an event-driven architecture using Apache Kafka and Kafka Streams. It highlights the challenges of the previous version, including lack of persistence, inability to scale horizontally, data loss, and performance bottlenecks due to chained joins and large stateful aggregates. The new design introduces finer-grained Kafka topics (Handshakes, Identifications, Verification, Session results) and initially uses Kafka Streams for stream processing, later incorporating ksqlDB for specific tasks. The post also discusses the 'false start' with a Session Aggregator approach that led to memory issues and rebalancing problems, and the subsequent architectural improvement focusing on multiple independent and parallel pipelines with single logical responsibilities to address these issues.