Real-Time Monitoring with Apache Kafka and Kafka Connect
Aug 17, 2022Simon Aubury

Summary
Introduces a proof-of-concept for wildlife monitoring using edge devices (Raspberry Pi) with TensorFlow Lite for real-time animal detection. Implemented a Kafka producer using confluent_kafka API to publish detection events to Confluent Cloud. Utilized ksqlDB to stream process raw detection data, extracting animal counts and applying a 30-second tumbling window with MAX aggregation for smoothing observations. Leveraged Kafka Connect Elasticsearch connector to ingest processed data for visualization in Kibana and implemented an alerting mechanism using the Kafka Connect HTTP Sink Connector to trigger a Telegram bot for rare animal sightings.