Real-Time Presence Detection at Scale with Apache Kafka on AWS | Confluent

What this post added

This post details Zenreach's implementation of a real-time presence detection platform using Kafka Streams. It describes their use case of detecting walk-ins, enriching them with contact information, and maintaining walk-in counts. The post highlights the migration from Python scripts and Spark Streaming to Kafka Streams due to operational complexity and cost. It outlines the four services (Walk-In, DeviceToContact, Enriched Walk-In, Walk-In Count) and their roles in processing data. Key technical lessons learned include scaling Kafka Streams services by adjusting `num.stream.threads`, managing stateful services with persistent storage (EBS volumes) to ensure state availability during deployments, and improving fault tolerance by configuring `num.standby.replicas` to reduce recovery time after instance failures.