TechPrufer
Get started
Confluent
Confluent logoConfluent · Feature trail

Real-time Edge Messaging Integration

2 posts · Mar 2019 – Apr 2022

What this trail covers

This feature thread tracks the evolution of integrating edge devices with data streaming platforms for real-time data ingestion and processing. It covers architectures for connecting IoT devices to cloud platforms like AWS IoT Core and Confluent Cloud, enabling contextualization of raw device data by combining it with business system information. The goal is to enrich data streams for improved decision-making, operational efficiency, and enhanced customer experiences. This includes configuration and bidirectional communication between edge devices and Kafka topics via PubNub's Edge Messaging Platform, enabling direct connectivity from web and mobile apps to Kafka without complex gateway infrastructure.

All Confluent trails
  1. Building a Real-Time Betting Platform with Confluent Cloud and AblyThis post details the architecture and implementation of a real-time betting platform by combining Confluent Cloud with Ably. It highlights the use of the Ably Kafka Connector for bidirectional data flow between Confluent Cloud topics and Ably. The architecture involves using Confluent Cloud for backend event streaming and processing (including ksqlDB for materialized views and odds generation), and Ably for distributing real-time odds and notifications to end-users via WebSockets, as well as collecting bets from users back into Confluent Cloud. The post emphasizes the benefits of this integration, including low latency, scalability, reliability, data integrity, and ease of use.
  2. Event Streaming Across Networks and Corporate Firewalls Using PubNub and Confluent PlatformThis post introduces the PubNub Bridge for Apache Kafka with Confluent Platform, a beta integration that enables bidirectional communication between PubNub's Edge Messaging Platform and Confluent Platform. It details how this bridge simplifies the process of driving real-world actions from Kafka events to mobile and web applications, bypassing the need for complex gateway APIs, firewalls, and IT involvement for opening ports. The post outlines use cases such as live alerts, mobile workforce transformation, BYOD environments, and personalization. It also provides a practical, hands-on demonstration using Docker Compose to set up Kafka, ZooKeeper, a sample feed, and the PubNub Bridge, showcasing direct connectivity from web and mobile apps to Kafka topics.
  3. Consuming Messages Out of Apache Kafka in a Browser | ConfluentThis post details the technical challenges and solutions for building a performant UI to visualize high-throughput Kafka message streams in a web browser. It explores the limitations of pagination and long polling, the benefits and initial performance issues of WebSockets, and the eventual adoption of Web Workers to offload processing and maintain UI responsiveness. The post also discusses optimizations made to state management (moving from Redux to React components) and the use of batching within web workers to further improve throughput.