Real-Time Presence Detection at Scale with Apache Kafka on AWS | Confluent

This post details Zenreach's migration from Python scripts and Spark Streaming to Kafka Streams for their real-time presence detection platform. It outlines their use case of detecting walk-ins, enriching them with contact information, and maintaining walk-in counts. The post highlights the challenges faced with a monolithic Spark application (deployment complexity, monitoring, cost) and the benefits of adopting Kafka Streams, including horizontal scalability via stream threads and tasks, persistent state management using RocksDB and EBS volumes, and improved fault tolerance with standby replicas. It also touches upon deployment on AWS ECS and integration with Prometheus and Rollbar.