Real-time Field Service Management Optimization

1/25/2024

This post details Confluent's internal use of Confluent Cloud to build a real-time field operations alerting system. It describes the challenges of translating events into actionable outcomes at scale, including event identification, capture from disparate systems, and enrichment with contextual data. The architecture leverages Confluent Cloud's managed connectors (e.g., Salesforce), Cluster Linking, and custom producers to ingest events. Kafka Streams is used for real-time event enrichment and filtering, with a specific example of joining connector failure events with account assignment data to notify the relevant team. Alerts are delivered via a Slack application. The post also mentions the potential to use Confluent's Flink offering for stream processing.

12/12/2023

This post details the implementation of a real-time field service management optimization solution for a telecommunication company. It describes the challenges of manual, non-real-time processes in managing third-party service providers and presents an event-driven architecture using Confluent Cloud. The solution involves modifying CRM systems to generate real-time events, feeding them into a 'Topic-as-a-Service' architecture, and routing them to third-party companies via HTTP sink connectors. Stream processing (Flink) is used for data enrichment, and Schema Registry ensures data quality. The post highlights business outcomes such as incident-free operations, improved collaboration, enhanced efficiency, scalability, and increased revenue.

10/17/2023

This post details building a real-time inventory system for retail using Confluent Cloud. It covers consuming data from various sources (in-store and e-commerce databases) using CDC connectors, processing this data with ksqlDB to create streams for sales and receipts, and then aggregating these into an 'inventory' stream. A ksqlDB table 'stock_on_hand' is constructed to provide a real-time, queryable view of inventory levels per SKU. The post also discusses distributing this real-time inventory data to systems like MongoDB for enhanced fulfillment decisions and customer-facing applications, highlighting the use of over 120 pre-built connectors and stream processing tools like Kafka Streams and Flink.

7/21/2023

This post details ACERTUS's implementation of real-time order notifications by migrating from siloed, batch-processed data systems to a microservices architecture using Confluent's event streaming platform. It highlights the technical challenges of integrating multiple databases (Postgres, MySQL, SQL Server) and the adoption of Confluent's CDC connectors, ksqlDB for stream processing and microservice development, and Stream Governance features (Schema Registry, Stream Lineage). The post emphasizes Confluent's advantages over alternatives like Kinesis and MSK, particularly its infinite storage, managed connectors, and ksqlDB capabilities, enabling ACERTUS to provide instant customer updates and improve operational efficiency.

6/27/2023

Mobile Premier League (MPL) migrated from a batch pipeline system to Confluent Cloud to enable real-time data processing for improved player matchmaking, faster decision-making, and real-time fraud detection. They process hundreds of millions of events daily, including client and server events from their gaming app, relational databases, and marketing SaaS apps. This enables personalized in-app offers, customized onboarding journeys, and real-time anomaly detection for fraud prevention, leading to reduced player churn and increased operational efficiency.

4/15/2021

This post details the development of a custom stateful connector that bridges Apache Kafka event streams with the Camunda workflow engine for a vehicle maintenance use case. It addresses challenges in translating raw sensor data into actionable insights by implementing solutions for duplicate event prevention, time windowing (using ksqlDB or Kafka Streams), and semantic aggregation. The custom connector, built with Java and Kafka/Camunda clients, maintains a persistent history to enable semantic aggregation and stream replay, facilitating the triggering of workflow instances (e.g., scheduling maintenance) or cancellation based on event streams.

11/23/2020

This post details Boden's digital transformation journey using Confluent Cloud and Apache Kafka. It highlights the migration from a legacy monolith to a microservices architecture to address integration challenges and the need for near real-time data. Key technical implementations include using Kafka for exposing product data changes from a PIM system, avoiding point-to-point integrations, and leveraging Schema Registry for event schema versioning. The post also describes using ksqlDB to aggregate clickstream data and feed it into Snowflake for business analytics, enabling insights into customer journeys and buying preferences. It also touches upon establishing standards for business events and architecture patterns for domain services, including aggregation services.

7/29/2020

This post details the implementation of Change Data Capture (CDC) using Debezium and Kafka Streams to ingest data from MongoDB into Kafka. It addresses the challenge of handling partial updates from the Debezium MongoDB CDC Connector by using Kafka Streams to maintain a table of complete documents. The post explains the concepts of streams and tables in Kafka Streams, provides a Java code example for a Kafka Streams application that performs a left join between a CDC stream and a table to merge changes, and details the merge function logic for handling create, read, update, and delete operations.