TechPrufer
Get started
Confluent
Confluent logoConfluent · Feature trail

Edge Data Ingestion and Processing

3 posts · Apr 2019 – Nov 2024

What this trail covers

Confluent Cloud continues to evolve its capabilities for integrating edge devices with data streaming platforms for real-time data ingestion and processing. This includes architectures for connecting IoT devices to cloud platforms like AWS IoT Core and Confluent Cloud, enabling contextualization of raw device data by combining it with business system information. The goal is to enrich data streams for improved decision-making, operational efficiency, and enhanced customer experiences. This includes configuration for edge deployments.

All Confluent trails
  1. Bridge AWS IoT Core and Confluent Cloud to Contextualize Your IoT DataThis post details the technical steps to integrate AWS IoT Core with Confluent Cloud for contextualizing IoT data. It covers setting up Confluent Cloud with API keys and a Kafka topic, configuring AWS Secrets Manager to store Confluent credentials, creating an AWS IoT Core VPC destination, and setting up an AWS IoT Core rule to route messages from the 'device_data' MQTT topic to the Confluent Cloud 'device_data' Kafka topic. It also includes optional debugging steps using error actions and AWS IoT Core logging.
  2. Unlocking the Edge: Data Streaming Goes Where You Go with ConfluentThis post details how Apache Kafka is being adopted at the edge for use cases requiring reduced latency, mitigation of networking limitations, cost optimization, and enhanced security. It highlights Kafka's ability to act as a local buffer for disconnected or intermittent networks, with Confluent's Cluster Linking technology ensuring data ordering consistency when replicating to the cloud. The post also discusses how stream processing with Kafka can optimize costs by filtering or compressing data sent to the cloud, and mentions Confluent for Kubernetes and Confluent Ansible as deployment and operational tools for managing edge Kafka clusters.
  3. The Role of Data Streaming in Smart Cities | Confluent for IoTThis post details how Confluent's data streaming platform, leveraging Apache Kafka and Apache Flink, can be used to build smart city applications. It highlights the integration of various data sources (CCTV, crowd-sourced reports, weather, vehicle position) via MQTT and an MQTT Source connector into Confluent Cloud. Apache Flink is used for real-time stream processing to calculate congestion levels and determine optimal routes for emergency vehicles. The post emphasizes Confluent's broad integration capabilities with over 80 connectors, low latency processing powered by the Kora engine, and resilience through Kafka's distributed architecture.
  4. Confluent at a Fully Disconnected Edge | Confluent Deploying Apache Kafka at the Edge with Confluent and AWSThis post details the implementation of Apache Kafka at the edge using Confluent Platform and AWS Snowball Edge for disconnected environments. It outlines the preparation steps, including deploying the S3 source connector (using AWS Fargate for a serverless experience) and creating a custom AMI for Confluent Platform on Snowball Edge. The deployment process involves ordering a Snowball Edge device with the custom image and field deployment. It also discusses data processing at the edge using ksqlDB and the rotation scheduling for data syncing to the cloud via S3.
  5. From the Cellar to the Cloud – How Aedifion is Driving Next-Generation Building Automation with Confluent | Confluent Smart Buildings with Real-Time IoT Analytics Using Confluent and AedifionThis post details aedifion's implementation of a data pipeline for building automation using Confluent Cloud. It outlines the requirements for scalability, security, and availability in collecting data from diverse building automation systems. The company chose Apache Kafka as their messaging system and Confluent Cloud for its managed service, citing its pay-per-use pricing, integrations (e.g., Postgres, MQTT), simplicity, Schema Registry capabilities, and on-premise deployment option. The setup involves edge devices publishing data via Kafka to Confluent Cloud, using the REST Proxy for HTTP-only firewalls, and employing stateless connectors for data ingestion into SQL and NoSQL databases. They highlight the benefits of Kafka as a buffer and replay mechanism, and the minimal operational overhead of Confluent Cloud.
  7. IoT Reference Architecture and Implementation Guide Using Confluent and MongoDB Realm | Confluent Build Your Own IoT Platform with Confluent and MongoDBThis post details a reference architecture and implementation guide for building an IoT pipeline using Confluent and MongoDB. It presents two solution architectures for thick and thin edge use cases, illustrating data flow from sensors to Confluent Platform (at the edge or in the cloud) and then to MongoDB Atlas. A specific cash logistics use case is elaborated, demonstrating the implementation with Confluent for data ingestion, ksqlDB for data transformation (including geospatial data preparation), and MongoDB Atlas (with time series and geospatial collections) for data storage. It also highlights the integration with MongoDB Realm for mobile data synchronization and MongoDB Webhooks for analytical purposes. Connector configurations for MongoDB Atlas Sink are provided.
  8. Modernize Apps and Infrastructure with Anthos, Confluent, and KafkaThis post details how Google Anthos and Confluent Platform can be combined to modernize applications and infrastructure, particularly for hybrid and edge deployments. It highlights the use of Confluent Platform on Anthos for consistent data infrastructure across environments, enabling event-driven architectures and microservices refactoring. The post specifically discusses edge manufacturing scenarios where Kafka and Kafka Streams are used for real-time data processing, anomaly detection, and data curation before sending curated streams to Confluent Cloud. It emphasizes Anthos's role in managing applications and infrastructure at the edge, providing persistent storage and stream processing capabilities for edge sites with inconsistent connectivity.
  9. Confluent, MQTT, and Apache Kafka Power Real-Time IoT Use CasesThis post introduces Waterstream, a Kafka-native MQTT broker that integrates directly with Apache Kafka. It details how Waterstream allows Kafka clusters to function as full-featured MQTT brokers, handling MQTT client connections, subscriptions, and message persistence directly within Kafka. The post highlights the benefits of operating a single distributed infrastructure for both Kafka and MQTT, reducing operational overhead. It also discusses the trade-offs of different MQTT integration options with Kafka, including Kafka Connect connectors and dedicated MQTT brokers, emphasizing Waterstream's Kafka-native approach for large-scale IoT use cases.
  10. IoT and Event Streaming at Scale with Kafka & MQTTThis post details the integration of IoT devices with Apache Kafka and its ecosystem, specifically focusing on the use of MQTT. It outlines common IoT use cases and the challenges of integrating edge devices with enterprise systems. The post presents three Kafka-native options for MQTT integration: Kafka Connect source/sink connectors, Confluent MQTT Proxy, and Confluent REST Proxy. It highlights the pros and cons of MQTT and Kafka individually, and explains how their combination addresses the limitations of each for building scalable, reliable, and secure IoT infrastructures. The post emphasizes the role of Kafka Connect in pulling data from MQTT brokers at its own pace and the benefits of Kafka's stream processing, long-term storage, and enterprise integration capabilities for IoT data.
  11. Scylla and Confluent Integration for IoT Deployments | ConfluentThis post introduces an end-to-end IoT architecture pattern combining MQTT, Apache Kafka, and ScyllaDB. It details how MQTT is used for device communication and lightweight messaging, while Kafka handles high-velocity data ingestion. ScyllaDB is integrated via Kafka Connect as a scalable NoSQL store for time-series data. The post provides a demo walkthrough using Confluent MQTT Proxy, Mosquitto for message simulation, and the Kafka Connect Cassandra connector (which works with ScyllaDB) to store data in Scylla.
  12. Baader's IoT and Data Streaming Innovation in Food Value ChainThis post details how BAADER is using Confluent Cloud and Apache Kafka to build a data-driven food value chain, capturing data from farm to fork. It highlights the use of Confluent Cloud for massive-scale, real-time IoT event ingestion and processing, enabling improved animal welfare, truck tracking, and predictive maintenance. The post also mentions Mojix's retail IoT platform, which uses Confluent Cloud to ingest data from various sensors for real-time inventory accuracy. Key technical aspects discussed include the scalability and reliability of Confluent Cloud for handling diverse IoT data formats, the use of Kafka Connectors, KSQL, and Kafka Streams for analysis and event processing, and the ability to stream across hybrid and multi-cloud environments. BAADER specifically leverages KSQL for processing and connectors for visualization with Elasticsearch and Kibana, and shares insights with partners. The post emphasizes offloading Kafka management to Confluent experts and the ease of use of KSQL for streaming applications.