Real-Time Risk Management Using Apache Kafka and Event Streaming

What this post added

This post details the technical evolution of Tesco's Risk Engine from a JMS-based system to an event-driven architecture using Apache Kafka and Kafka Streams. It outlines the initial challenges with JMS, including lack of scalability, persistence, and concurrency. The rebuild introduced a new event-driven design with finer-grained topics (handshakes, identifications, verification, session results) and leveraged Kafka Streams for stream processing. The post details the initial 'false start' with a Session Aggregator, highlighting issues with high memory demand, state management (RocksDB cache), frequent rebalancing due to lack of cooperative rebalancing, and data skew caused by large 'elephant' aggregates. Remediation efforts included limiting aggregate size and scaling, but ultimately a redesign was necessary. The improved architecture emphasizes multiple independent and parallel pipelines, avoiding a monolithic aggregate, and focusing on specific goals like blocking malicious IP addresses. The post discusses the technical trade-offs and lessons learned during the migration and redesign process.