IoT Fleet Management Data Streaming

11/20/2023

This post details the technical challenges of using MQTT for IoT fleet management, specifically its limitations in stream processing and data reprocessing. It proposes a solution by integrating MQTT with Apache Kafka, leveraging Confluent's MQTT Source and Sink connectors, stream processing capabilities (ksqlDB), and Schema Registry to build a real-time fleet management system. The architecture involves streaming telemetry data, processing it in real-time, and syncing it to downstream systems like MongoDB Atlas.

9/19/2023

This post details a specific customer's migration from self-managed Kafka on Azure HDInsight to Confluent Cloud to enable same-day delivery. It highlights the challenges of managing Kafka in HDInsight, including lack of expertise, missed upgrades, frequent pipeline failures, limited connector availability, and no stream processing capabilities. The solution implemented uses Confluent Cloud's managed service, multi-AZ dedicated clusters with VNET peering, pre-built connectors (Debezium, JDBC Sink), and Schema Registry. The company also utilized the .NET client library. The outcome is improved reliability, reduced maintenance, and increased innovation capacity.

10/27/2022

This post details the construction of a real-time IoT motion detection and alerting system using Confluent Cloud and ksqlDB. It outlines the hardware (Raspberry Pi Zero W) and software stack, including the use of the Motion software for detection, cURL for AWS S3 uploads, librdkafka for publishing events to Confluent Cloud, and Docker for containerization. The architecture involves IoT devices emitting motion events to Kafka, which are then processed by ksqlDB. The post demonstrates ksqlDB stream creation and querying for both real-time monitoring (using push queries with EMIT CHANGES) and past event search (using pull queries). It also includes code snippets for the ksqlDB push query web worker and a sample SQL pull query.

8/17/2022

This post details a hackathon project demonstrating real-time wildlife monitoring using Apache Kafka and Kafka Connect. It covers using TensorFlow Lite on a Raspberry Pi for object detection, publishing events to Kafka via the confluent_kafka Python client, processing events with ksqlDB (including sliding window aggregation and JSON field extraction), and visualizing data in Kibana via the Kafka Connect Elasticsearch connector. It also includes setting up a Telegram bot with a Kafka Connect HTTP Sink Connector for rare event alerting (teddy bear sightings).

8/2/2022

This post details building an end-to-end IoT alerting system for houseplants using Confluent Cloud and ksqlDB. It covers hardware setup with Raspberry Pi and I2C capacitive soil moisture sensors, data ingestion into Kafka topics ('houseplants-readings', 'houseplants-metadata') using a Python producer, and stream processing with ksqlDB to determine watering needs. The post also discusses the choice of Kafka over MQTT for stream processing and persistence, and the importance of message structuring and topic design.

5/4/2022

This post details how Aedifion uses Confluent Cloud for building automation and IoT analytics. It covers their requirements for scalability, security, and availability, their choice of Apache Kafka and Confluent Cloud, and the setup of their data pipeline. Key technical aspects include using Confluent REST Proxy for HTTP-only environments, a strict topic naming schema for access control and tenant separation, and self-built stateless connectors for ingesting data into SQL and NoSQL databases. The post highlights the advantages of Kafka as a buffer and its replay capabilities, and mentions a minor hurdle with MQTT-native hardware while awaiting Confluent's MQTT Proxy.

10/23/2020

This post demonstrates how to build an Intrusion Detection System (IDS) using ksqlDB and Kafka. It details setting up a virtualized network environment with Vagrant, capturing network packets using TShark, and ingesting them into Kafka via the Spool Dir connector. The post then shows how to use ksqlDB to process these raw network packets, flatten the JSON data, and identify potential port scan and Slowloris attacks by creating separate ksqlDB streams for each attack type. It also provides example ksqlDB queries for data analysis and detection.

7/16/2020

This post details building a real-time asset tracking system for public transport using Apache Kafka, Kafka Streams, MQTT, Elasticsearch, and Kibana. It covers the architecture for ingesting geolocation data from an MQTT server into Kafka, enriching the data using Kafka Streams to create a nested JSON structure, and then sinking it into Elasticsearch using the Elasticsearch Sink Connector for visualization in Kibana. Specific technical details include Dockerizing the Java producer, configuring Kafka topics with partitions, setting up Elasticsearch dynamic templates for geo_point mapping, and configuring the Kafka Connect Elasticsearch Sink Connector.

7/15/2020

This post introduces Waterstream, a Kafka-native MQTT broker that runs as a Kafka Streams application, allowing direct integration of MQTT devices with Kafka. It details how Waterstream stores MQTT state (subscriptions, QoS status, retained messages) in Kafka and enables bidirectional communication. The post also provides a comprehensive overview of various MQTT integration options with Kafka, including Kafka Connect connectors, Confluent MQTT Proxy, vendor-specific integrations, and the Kafka-native approach, discussing their respective pros, cons, and considerations for large-scale use cases.

6/5/2020

This post showcases how to build a simple fleet management solution using Confluent Cloud, ksqlDB, Kafka Connect with MongoDB connectors, and MongoDB Atlas. It details acquiring telemetry data from fleets in real-time using Voluble, processing and acting on real-time events (e.g., hazard events), and correlating multiple events for use cases like delivery ETA calculation. It also covers storing events and location data in MongoDB Atlas for historical analysis and managing driver/fleet lifecycle data.

6/4/2020

This post details best practices and techniques for stream processing with IoT data, focusing on challenges like variable connectivity, bursty data, long tails of firmware versions, DoS-like events, mixed service levels, and varying data formats. It highlights Apache Kafka as the core technology for durable storage, horizontal scalability, high throughput, and low latency. The post discusses raw data ingest strategies using intermediaries like web servers or MQTT endpoints, and addresses handling large messages from disconnected devices by batching and careful partitioning strategies based on epoch milliseconds. It also touches upon the need for different storage and processing mechanisms for varying message priorities and durability requirements.

5/12/2020

This post details building a Telegram bot powered by Kafka and ksqlDB. It demonstrates how to ingest Wi-Fi packet data into Kafka, process it with ksqlDB to create stateful aggregates (tables), and then query these aggregates using pull queries to serve on-demand information to a Telegram bot. The technical depth includes ksqlDB schema declaration, table creation with aggregations (COUNT, MIN, MAX, COLLECT_SET), and the use of interactive queries (via REST API) to access the materialized state. It also touches upon the underlying storage mechanisms of ksqlDB (Kafka changelog, RocksDB) and the concept of events as the primary system of record.

5/22/2019

This post details an end-to-end IoT architecture pattern integrating MQTT, Apache Kafka, and ScyllaDB. It explains how MQTT is used for device communication and lightweight messaging, while Kafka handles high-velocity data ingestion. ScyllaDB is used as a persistent, scalable NoSQL store for long-term data storage. A demo is provided showcasing the use of Confluent MQTT Proxy, Kafka Connect Cassandra connector, and Mosquitto for simulating sensor data and streaming it into ScyllaDB.

4/25/2019

This post details how BAADER is using Confluent Cloud to build a data-driven food value chain by capturing IoT event data from farm to fork. It highlights the use of KSQL for real-time processing of machine efficiency data and delivery truck GPS data, integrating with external APIs, and alerting factory managers and customers. BAADER is also collecting weather data to correlate with GPS data for product quality and animal welfare insights. The post emphasizes Confluent Cloud as a fully managed Kafka service and a single source of truth for event data, enabling BAADER to focus on application development rather than infrastructure management.

4/18/2019

This post details BAADER's use of Confluent Cloud and Apache Kafka to build a data-driven food value chain, capturing data from farm to fork. It highlights the use of Confluent Cloud for its scalability, reliability, and connectors for integrating disparate data sources. The post also mentions Mojix's retail IoT platform using Confluent Cloud for real-time inventory management. Key technical aspects include using Kafka Connect, KSQL, and Kafka Streams for collecting, transporting, storing, and processing IoT events globally. It emphasizes Confluent Cloud's ability to handle massive data volumes, provide real-time analytics, and support hybrid/multi-cloud environments. BAADER specifically uses KSQL for stream processing and connectors to Elasticsearch and Kibana for visualization.

10/18/2018

This post details using a Raspberry Pi with a Software-Defined Radio (SDR) to capture ADS-B aircraft transmissions. The captured data is then sent to Kafka topics (location-topic and ident-topic) via the Confluent REST Proxy. KSQL is used to enrich this data by joining it with reference data (ICAO to aircraft mapping and callsign details) to create new streams (location_and_details_stream and ident_callsign_stream). These enriched streams are then pushed to Elasticsearch using Kafka Connect for visualization in Kibana dashboards. The post demonstrates a specific use case of identifying a low-flying aircraft that woke the author's cat by querying the enriched location data.

4/13/2018

This post details how to use the Confluent Kafka Python client and KSQL to build an event-driven alerting system. It demonstrates filtering syslog data in real-time with KSQL to identify suspicious login attempts and then using a Python script to consume these filtered events from Kafka and send push notifications to Slack. The post covers installing the Slack and Confluent Kafka Python clients, consuming messages from Kafka, and integrating the two to send Slack notifications based on Kafka topic messages.