TechPrufer
Get started
Confluent
Confluent logoConfluent · Feature trail

Real-time Insurance Quoting with Data Streaming

6 posts · Oct 2018 – Sep 2024

What this trail covers

This feature thread tracks the evolution of real-time order notifications and logistics management. It details how companies have migrated from siloed, batch-processed data systems to modern microservices architectures powered by Confluent's event streaming platform. Key technical aspects include integrating disparate databases (Postgres, MySQL, SQL Server) using CDC connectors, leveraging ksqlDB for real-time data processing and microservice development, and utilizing Stream Governance for data.

All Confluent trails
  1. Accelerating Insurance Claims With a Stream Processing ArchitectureThis post details a prototype for real-time auto claims processing using Confluent Cloud and Apache Flink. It outlines a 5-step event-driven architecture: streaming new claims (FNOL) to a Kafka topic, enriching claims with customer and policy data via temporal joins in Flink (using Oracle CDC connector), filtering and routing claims with Flink for initial validation, routing for straight-through processing or human assessment, and finally settling the claim and sending notifications. The post highlights the benefits of temporal joins in Flink for accurate data enrichment and the use of managed connectors for data ingestion.
  2. Transforming T+1 Trade Settlement for Modern Markets With ConfluentThis post details how Confluent's data streaming platform can enable T+1 trade settlement by migrating from batch processing to real-time event streaming. It highlights the use of Confluent connectors to integrate disparate systems, the scalability and reliability of the Kora engine for market volatility, and the application of stream processing for data enrichment and delivery to siloed processes. It also touches on RBAC for enforcing entitlements and standards.
  3. Data Streaming in Healthcare: Achieving the Single Patient ViewThis post details how healthcare organizations can build a Single Patient View (SPV) using data streaming to address interoperability, regulatory compliance, and scalability challenges. It highlights the use of Apache Kafka and Confluent Cloud to process sensitive patient data securely and at scale, enabling use cases like real-time experimental treatment matching for cancer patients. The architecture involves ingesting EMR files, processing them with Flink, and syncing data to MongoDB Atlas for historical analysis, offering benefits like reduced intervention, HIPAA readiness, and simplified architecture.
  4. Real-Time Order Management: The Key to Streamlining Your Business OperationsThis post details how Confluent Cloud can be used to build a real-time order management platform. It highlights the use of Confluent's PostgreSQL and Oracle CDC Source Connectors to capture data from source databases, ksqlDB/Flink for real-time transformation and enrichment of order data, and the Lambda Sink Connector for sending order confirmations. It also mentions using HTTP Sink Connectors for integrating with third-party fraud prevention systems and validating order flooding. The post includes example ksqlDB queries for data enrichment and order validation.
  5. Real-Time Inventory in Retail with ConfluentThis post details a specific retail use case for real-time inventory management. It demonstrates how to ingest sales and receipts data from disparate sources using Confluent database source connectors and process it with ksqlDB to create a real-time `stock_on_hand` table. The post provides ksqlDB DDL and DML examples for creating streams and tables, and explains how this real-time inventory data can be consumed by downstream systems like order management, allocation, and merchandising applications to improve customer experience and operational decisions.
  7. The Role of Real-Time Interoperability to Healthcare PayersThis post details how healthcare payers can leverage real-time data streaming with Confluent and Apache Kafka to address challenges in interoperability. It outlines the importance of interoperability for payers, including improved care coordination, efficiency, data analytics, fraud prevention, and member experience. The technical challenges of siloed data and legacy communication methods are discussed. An example architecture is presented, demonstrating Kafka Produce-Consume patterns with Trizetta Facets and Change Data Capture (CDC) from Oracle databases. The post emphasizes how data streaming enables faster claims management, better member experiences, and reduced fraud risk.
  8. Providing Real-Time Insurance Quotes via Data StreamingThis post details how an Insurtech company migrated from a monolithic database architecture to a streaming data pipeline using Confluent Cloud. It highlights the use of Change Data Capture (CDC) connectors, specifically the Debezium PostgreSQL Connector, to stream row-level changes from a legacy database into Kafka topics. This enables downstream microservices to consume real-time data for applications like generating insurance quotes. The post also discusses future possibilities of using the platform for IoT and GenAI workloads and the integration of sink connectors to downstream systems like Elasticsearch, Snowflake, and cloud storage.
  9. Acertus Real Time Order NotificationsThis post details how ACERTUS migrated from siloed, monolithic data systems relying on batch ETL to a modern microservices architecture powered by Confluent's event streaming platform. Key technical contributions include integrating disparate databases (Postgres, MySQL, SQL Server) using Confluent's CDC connectors, leveraging ksqlDB for real-time data processing and microservice development, and utilizing Confluent's Stream Governance package (Schema Registry, Stream Lineage) for data management. The post highlights the benefits of Confluent's infinite storage, managed connectors, and ksqlDB for faster microservice development, enabling real-time order notifications and improved customer experience.
  10. How BigCommerce Uses Data Streaming to Bring Real-Time Insights to MerchantsBigCommerce migrated from a Hadoop MapReduce-based system and self-managed Kafka clusters to fully managed Confluent Cloud for real-time merchant analytics. The migration involved a phased approach, pumping data into both systems simultaneously to ensure zero downtime and data loss. Confluent Cloud provided a cloud-agnostic platform, immediate Kafka version and security upgrades, and reduced operational overhead for their data team. This enabled them to deliver real-time analytics and insights to merchants, handling traffic spikes automatically and seamlessly.
  11. Q&A – How Data Processing Pipelines Accelerate Loan OriginationThis post details how Mr. Cooper, a mortgage company, has adopted a streaming-first approach to accelerate loan origination and improve customer experience. They leverage Apache Kafka as their central nervous system for real-time eventing, using Kafka Streams and ksqlDB for stream processing, including real-time change comparison and delta calculation. The company has over 300 services in production and nearly 100 connectors. They also discuss their multicloud strategy, migrating from Azure to GCP using Confluent's Cluster Linking and Schema Linking for seamless data migration without downtime. The post emphasizes the importance of a mindset shift towards real-time data processing and recommends identifying high-impact use cases to drive adoption.
  12. How Change Data Capture (CDC) WorksThis post introduces Change Data Capture (CDC) as a pattern to extract data changes from relational databases into event streams. It explains the limitations of traditional databases for real-time reactions and contrasts CDC with older active database concepts like ECA rules and triggers. The post highlights the benefits of sourcing CDC events from transaction logs (e.g., using Debezium) over polling or trigger-based approaches. It also discusses the importance of denormalizing data for event streams and provides a conceptual example of how normalized order data can be transformed into a denormalized event.
  13. Sync Databases and Remove Data Silos with CDC & Apache KafkaThis post details the technical advantages and limitations of using JDBC vs. CDC for moving data from databases to Kafka. It provides a detailed example of a legacy bank using Debezium CDC connectors for PostgreSQL and SQL Server to synchronize two databases, and crucially, explains how to prevent infinite loops using Single Message Transforms (SMTs) with filtering based on a source column ('SRC'). It also references available source code for connectors and SMTs.
  14. Digital Transformation in Financial Services Using ConfluentThis post discusses how financial services organizations are leveraging data streaming platforms and event-driven architectures, specifically Apache Kafka via Confluent Cloud, to transform their businesses. It highlights use cases such as personalization of 'Know Your Customer' initiatives, real-time analytics and fraud detection, cybersecurity and regulatory compliance, trading platform and market data services, and customer tracking. The post emphasizes the benefits of modern data platforms and event-driven architectures in overcoming the challenges posed by increasing data volumes and regulatory requirements in the financial services industry.
  15. Database Streaming with Confluent and Amazon AuroraThis post details a specific migration pattern from an on-premise MySQL database to Amazon Aurora using Confluent Cloud. It outlines the steps involved in setting up Confluent MySQL Sink and Source connectors to synchronize data, enabling a cutover to the cloud-native database. The post emphasizes the use of ksqlDB for real-time data processing and the benefits of migrating legacy databases to managed cloud services.
  16. Risk Management in Volatile Markets with Data Streaming and AnalyticsThis post details the integration of Confluent Cloud and Databricks for real-time risk management in financial markets. It outlines an architecture that uses Confluent's streaming capabilities to ingest and enrich market and portfolio data, feeding it into Databricks' Lakehouse for advanced risk calculations. Key components include the SSE Source Connector for market data, ksqlDB for real-time enrichment and joining with portfolio data, and the Databricks Delta Lake Sink Connector for exactly-once semantics data writing. Delta Live Tables are used for incremental transformations and Monte Carlo simulations, with results visualized through Databricks SQL.
  17. How Bolt Adopted Change Data Capture with Confluent for Real-Time Data & AnalyticsThis post details Bolt's adoption of Change Data Capture (CDC) with Confluent Platform for real-time data and analytics. It explains the shortcomings of their previous batch data replication methods and outlines the requirements for a new pipeline, including linear scalability and proprietary data transformations. The article highlights the benefits of log-based CDC, the role of Debezium for capturing data changes, Apache Kafka as the core event streaming platform for ordered and replayable data, Kafka Connect for orchestrating CDC processes, Confluent Schema Registry for schema management and data contracts, and Kafka Streams/ksqlDB for stream processing and data transformation.
  18. Implement CDC & Streaming Analytics Using Kafka & DebeziumThis post introduces the use of Debezium MongoDB CDC connector to capture record-by-record changes from MongoDB and stream them to Kafka. It addresses the challenge of handling patch-based updates from Debezium by leveraging Kafka Streams to maintain a table of complete documents. A Kafka Streams application is provided that uses a left join between a stream of changes and a table to merge patches, materializing the result as a new stream of complete records. This ensures new consumers receive up-to-date, complete documents without needing to maintain state or merge logic themselves.
  19. Leverage Real-Time Clickstream Data Using ksqlDB and ElasticsearchThis post details building an end-to-end IoT motion detection and alerting system using Confluent Cloud and ksqlDB. It covers hardware choices (Raspberry Pi Zero W), software components (Motion, librdkafka, Docker), and the architecture for ingesting motion events into Kafka. The system utilizes ksqlDB for real-time stream processing and querying, enabling both real-time monitoring with push queries and historical event search with pull queries. The UI is built with React and Next.js, demonstrating how to leverage real-time clickstream data using ksqlDB and Elasticsearch for analysis and visualization.
  20. ATM Fraud Detection with Apache Kafka and KSQL | ConfluentThis post demonstrates how ksqlDB can be used to build a real-time fraud detection system for ATM transactions. It highlights the use of stream-stream joins with time windows to identify suspicious activity based on account number, location, and transaction timing. The post also shows how to integrate with Elasticsearch and Kibana for data storage and visualization, and discusses the nuances of stream-stream joins in KSQL, including the need for explicit windowing and the handling of self-joins.