Delivering Real-Time Personalization with Databricks and Redis | Redis

This post details the integration of Databricks Real-Time Mode (RTM) with Redis for real-time personalization. RTM is introduced as a continuous processing engine for Apache Spark's Structured Streaming, enabling sub-second performance with strict p99 latencies. The post explains how RTM processes clickstream events, maintains per-user session state (recency-weighted scores), and scores candidate products. The output is then written to Redis using a ForeachWriter, storing recommendations as a sorted set (product to score) and a hash (product title and price), both with short TTLs. The website reads this data from Redis in a pipelined round trip per render for sub-millisecond serving. Performance benchmarks show under 160ms p99 from click to recommendation served at 100k events/s. The post also outlines the broader applicability of this compute-and-serve pattern across various use cases like fraud detection, agent coordination, and inventory management.