TechPrufer
Sign inGet started

Blogs›Confluent›Edge Data Ingestion and Processing

Edge Data Ingestion and Processing

Edge Data Ingestion and Processing

3
posts
2019–2022

This post details building an end-to-end IoT motion detection and alerting system using Confluent Cloud and ksqlDB. It covers hardware choices (Raspberry Pi Zero W), software components (Motion, librdkafka, Docker), and the architecture for ingesting motion events into Kafka. The system utilizes ksqlDB for real-time stream processing and querying, enabling both real-time monitoring with push queries and historical event search with pull queries. The UI is built with React and Next.js, demonstrat

2022

Confluent at a Fully Disconnected Edge | Confluent Deploying Apache Kafka at the Edge with Confluent and AWS

5/12/2022

This post details the architecture and implementation for deploying Apache Kafka and Confluent Platform on AWS Snowball Edge devices for fully disconnected edge environments. It covers preparing a custom EC2 image with Confluent Platform and the S3 Sink connector, ordering and deploying Snowball devices for data collection, and using the S3 Source connector to ingest data into Confluent Cloud. The post also discusses leveraging ksqlDB for edge-based stream processing and provides an overview of data processing and application integration strategies.

IoT Reference Architecture and Implementation Guide Using Confluent and MongoDB Realm | Confluent Build Your Own IoT Platform with Confluent and MongoDB

1/26/2022

This post presents a reference architecture and implementation guide for building IoT pipelines using Confluent and MongoDB. It details two solution architectures: 'thick edge' for large industries and 'thin edge' for businesses with smaller devices. For thick edge, it describes aggregating data at the edge, pushing it to Confluent Platform using connectors (MQTT/REST), storing it transiently in Kafka for edge processing, and replicating it to Confluent Cloud. For thin edge, data is directly published to Confluent Cloud. A cash logistics use case is elaborated, demonstrating the implementation with Confluent Cloud, MongoDB Realm, time series collections, and database triggers. It includes ksqlDB queries for data transformation and connector configurations for writing data to MongoDB Atlas (both time series and geospatial collections). The post also highlights the use of MongoDB Realm Sync for mobile applications and MongoDB Webhooks for accessing time series data.

2019

IoT and Event Streaming at Scale with Kafka & MQTT

10/10/2019

This post details how Apache Kafka and its ecosystem (Kafka Connect, Kafka Streams, ksqlDB) are used for integrating and processing IoT data at scale. It specifically highlights Kafka-native options for MQTT integration, including Kafka Connect source and sink connectors, Confluent MQTT Proxy, and Confluent REST Proxy. The post discusses common IoT use cases, requirements, and challenges, contrasting MQTT's limitations with Kafka's strengths as an event streaming platform. It provides a high-level architecture for enterprise IoT integration and details the pros and cons of both MQTT and Kafka in this context.