TechPrufer
Get started
Confluent
Confluent logoConfluent · Feature trail

Real-time Enterprise RAG

67 posts · Jun 2018 – Sep 2026

What this trail covers

Confluent Cloud enhances real-time, personalized customer experiences by integrating in-store and online data for timely offers and product information. The architecture emphasizes high availability, scalability, and low latency using Kafka, ksqlDB, and notification platforms. This iteration focuses on powering AI agents with real-time context from across the business, enabling them to make accurate, situationally aware decisions and take actions. This includes Streaming Agents (Phase 2 - Open P, and AI Model Inference in Flink for real-time recommendations and personalized offers.

All Confluent trails
  1. Building an Agentic SOC on a StreamThis post details the engineering of an Agentic Security Operations Center (SOC) that autonomously investigates security alerts. It describes a novel architecture featuring a central triage coordinator, specialized evidence agents, an adversarial evaluator, and a self-learning knowledge base. Key technical contributions include filtering duplicate alerts, enriching alerts with institutional knowledge, dispatching specialist agents for deep queries, adversarial review by a secondary agent to validate conclusions, and a streaming architecture for prioritized alert handling. Agent telemetry is streamed back to Confluent Cloud for real-time context and monitoring.
  2. Hemut: Building the Internet of Freight on Real-Time DataHemut built its AI-native operating system for trucking on Confluent Cloud, establishing a real-time data foundation that supports its entire platform. The system has processed over 286 million events without loss, with position updates moving through the platform in approximately one-tenth of a second. Confluent's Schema Registry enforced data contracts during telematics provider onboarding and data format evolution, ensuring zero downtime and no breaking changes for downstream consumers. The use of Flink for stream processing on this foundation enables AI model retraining on historical data and real-time application development.
  3. New in Confluent Intelligence and AI Tools: Making Agents Native to the Stream, Expanded Model Support, New Agent Skills, and CopilotIntroduces expanded model support for time-series anomaly detection and forecasting with IBM Granite models (TTM, FlowState, PatchTST-FM) and Google TimesFM model, callable directly in Flink SQL via AI_FORECAST and AI_DETECT_ANOMALIES functions. Enhances the Real-Time Context Engine with user-facing metrics for usage, performance, health, and cost visibility. Adds lightning queries (Early Access) for low-latency REST API access to live state. Updates the fully managed MCP server with write tools for connector restart and update connector configuration, and Flink tools for creating and managing Flink SQL. Introduces new open-source Agent Skills for MSK migration, Java Kafka producer/consumer, Flink UDFs, and a Confluent Skill creator. Launches Confluent Copilot for Confluent Cloud (Early Access) for a hosted, natural-language experience.
  4. Customer Intelligence Hub for Real-Time GTM InsightsThis post introduces the Customer Intelligence Hub (CIH), an internal Confluent application designed to provide a single, prioritized view of each customer account by detecting changes and suggesting next-best actions. CIH centralizes signals from Salesforce, Zendesk, product telemetry, billing, Jira, and other systems into Apache Kafka topics. It uses Apache Flink for stream processing, normalization, enrichment, and generating semantic embeddings for AI use cases. Enriched facts and embeddings are streamed to OpenSearch for vectorization and search, enabling grounded GenAI responses (AccountIQ) that are scoped by account and user authorization. The architecture supports real-time answering of questions with backend authorization checks. Key capabilities include an Account Center for portfolio overview, a Whitespace View for expansion opportunities, a Real-Time Activity Feed for prioritized events, AccountIQ for GenAI-powered customer context, and automated Executive Account Summaries. The post also shares early adoption metrics and user feedback, highlighting CIH's integration into daily workflows for account management and customer success.
  5. RAG and GenAI for Customer 360 ArchitectureThis post details the architectural shift required for AI-powered Customer 360, moving from legacy batch-oriented data warehouses to real-time, event-driven systems. It highlights the limitations of stale data for GenAI and RAG, emphasizing the need for continuous event ingestion, unified real-time customer state, context-aware retrieval, and guardrailed generation. The architecture combines a real-time data engine (Kafka, Flink, Customer Profile Store), a knowledge and embedding pipeline (document sources, embedding generation), and a retrieval and AI layer (RAG, GenAI Copilot) with cross-cutting observability and governance. The data flow emphasizes bypassing traditional database latency by using an event streaming platform to update the live customer profile in milliseconds, fusing this state with curated domain knowledge for RAG, and ensuring safety and compliance through guardrailed prompt construction.
  7. Agentic Event-Driven Systems ArchitectureIntroduces the concept and architecture of agentic event-driven systems, defining them as autonomous event-driven architectures where AI agents continuously sense events, reason over shared state, take actions, and learn from outcomes in real-time without human-in-the-loop orchestration. Details a layered architecture comprising Event Producers, Streaming Backbone (e.g., Kafka), Stateful Stream Processing (e.g., Flink), Shared State & Context Layer, Agent Execution Layer (LLM-based, ML models, rules engines), Orchestration & Policy Engine, Command & Event Emission, and Observability & Governance. Contrasts this with traditional reactive event-driven systems, highlighting the shift from reactive to adaptive, autonomous decision-making and the architectural evolution required for AI scale.
  8. Enterprise Knowledge Management with RAGThis post details a production-grade Enterprise RAG architecture that leverages Confluent for event streaming as the backbone. It contrasts static knowledge bases with real-time RAG, highlighting architectural differences in ingestion, context synchronization, and scalability. The deep dive covers knowledge sources, event streaming ingestion via CDC, real-time processing with Flink for chunking and PII redaction, streaming embedding generation, real-time vector store upserts, retrieval with RBAC filtering, LLM generation, and an observability layer for data lineage. The real-time data flow is explained step-by-step, emphasizing how Confluent enables document ingestion, granular embedding updates, and cross-system context synchronization. Core capabilities like continuous embedding updates and cross-system context synchronization are elaborated with architectural enablers and real-world examples.
  9. RAG and GenAI for Regulated SectorsThis post details the architectural considerations for building a compliant RAG/GenAI system in regulated environments. It emphasizes the shift from simple database-to-LLM scripts to an event-driven architecture using Confluent for data streaming. Key contributions include outlining the challenges of sensitive data exposure, hallucination risk, auditability, data sovereignty, and policy enforcement. It proposes a compliant RAG architecture with governed ingestion, access-controlled retrieval, policy-aware generation, and auditable outputs. The post details a streaming RAG ML pipeline involving controlled ingestion via CDC, stream processing with PII masking and schema enforcement, secure vector indexing with context scoping, and an LLM layer with observability for response provenance. It contrasts real-time vs. batch RAG for compliance and lists risk mitigation strategies and business impacts.
  10. Agentic Fleet Management Architecture for Real-Time OperationsThis post introduces the concept and architecture of agentic fleet management, detailing a layered, event-driven system built on real-time data streams. It outlines core components from edge ingestion to command and control feedback loops, emphasizing autonomous decision-making agents, closed-loop orchestration, and continuous feedback cycles. The architecture leverages an event streaming backbone (e.g., Kafka) and stream processing (e.g., Flink) to enable dynamic route optimization, predictive maintenance, incident detection, and autonomous dispatch, shifting from reactive to proactive and self-optimizing fleet operations.
  11. Confluent Cloud Q2 2026: dbt adapter, Confluent Intelligence updatesIntroduces a managed Model Context Protocol (MCP) server for Confluent Cloud, providing a scalable and highly available interface for AI agents to discover and debug Confluent resources. Also introduces Confluent Agent Skills, which encode Confluent best practices directly into AI agents to accelerate development and improve deployment quality. Enhances Flink accessibility with a dbt adapter for Confluent Cloud, enabling data engineers to define, test, and deploy streaming pipelines using dbt workflows. Introduces Materialized Tables for simplified Flink pipeline evolution with automated offset management and job orchestration. Adds Process Table Functions (PTFs) for custom, stateful stream processing logic in Java. Enables external connectivity for user-defined functions (UDFs) to interact with external APIs and services. Announces General Availability for Snapshot Queries, unifying batch and streaming workloads by enabling point-in-time SQL queries on historical and real-time data.
  12. New in Confluent Intelligence: Real-Time Context Engine Upgrade, New Model Support, ML Functions, and MoreThe Real-Time Context Engine is now generally available with enhanced querying capabilities including filters, ranges, compound queries, projections, and ordering directly on live tables, reducing reliance on external databases. Streaming Agents and the Agent Management Console are also now generally available, offering production-ready, event-driven AI agents with enterprise-grade operations and a centralized UI. New ML functions for Multivariate Anomaly Detection (Open Preview), PII Detection (Early Access), and Sentiment Analysis (Early Access) have been introduced. Support for TimesFM (Early Access), Anthropic (GA), and Fireworks AI (GA) models has been added.
  13. Confluent's architectural approach to digital sovereignty for real-time streamingThis post introduces the concept of digital sovereignty as a core architectural requirement for real-time streaming, driven by evolving regulatory frameworks. It details three key ideas: 1) the distinction between policy assurances ('We will not') and architectural guarantees ('We cannot') with a focus on BYOC and BYOK patterns; 2) the schema as the new sovereignty boundary, enabling integrated data contract enforcement, sensitivity tagging, encryption directives (CSFLE/CSPE), and quality/policy rules; and 3) portability as a property of open protocols (Kafka, Flink) rather than vendor contracts, emphasizing the importance of open wire and on-disk formats for credible exit plans. It also discusses the operational costs of architectural sovereignty and provides a checklist for evaluating streaming architecture decisions.
  14. InfiniteWatch + Confluent: Turning Customer Interaction Data into Real-Time IntelligenceInfiniteWatch has integrated Confluent as the event backbone for its AI-native customer interaction intelligence platform. The platform ingests, buffers, processes, replays, enriches, and routes customer interaction events in real time. The architecture is designed to handle bursty, high-volume, and correlated data, enabling capabilities like burst absorption, durable event preservation, service decoupling, replayable intelligence, and multi-destination routing. The reference architecture follows a Capture -> Stream -> Enrich -> Understand -> Act flow, with Confluent topics used to stream signals from various sources to AI services and operational workflows.
  15. How Agent Taskflow Scaled AI Agents to 1M with Confluent and AWSThis post details how Agent Taskflow (ATF) scaled its AI agent platform to 1 million agents using Confluent Cloud and AWS. It highlights the architectural choices made to address the challenge of scaling event streaming infrastructure, specifically avoiding the operational burden of self-managing Apache Kafka. The solution leverages Confluent Cloud as the streaming backbone for all agent communications, enabling serverless agents that consume compute resources only when needed. Amazon Bedrock is used for LLM inference, ensuring low latency and data sovereignty within customer VPCs. Enterprise-grade security is achieved through IAM role-based authentication and Confluent's Schema Registry for data quality enforcement. Performance benchmarks are presented, demonstrating high-volume scalability, consistent sustained performance, infrastructure efficiency, ultra-low latency, predictable tail latency, zero-loss integrity, and battle-tested stability, even on minimal infrastructure. The post also outlines flexible deployment models for public SaaS and private enterprise environments.
  16. New in Confluent Intelligence: A2A, Multivariate Anomaly Detection, and MoreIntroduces Agent2Agent (A2A) Integration for Streaming Agents, enabling collaboration between agents on different platforms via Kafka. Adds Multivariate Anomaly Detection for Built-in ML Functions using Flink SQL's ML_DETECT_ANOMALIES_ROBUST. Expands Vector Search support to Azure Cosmos DB and Amazon S3 Vectors, integrating them into Flink's external tables and search fabric. Enables AWS and Azure Private Link for secure, private connectivity from Flink to external models, databases, and vector stores, keeping AI traffic off the public internet. Provides vendor-backed support for the open-source MCP Server.
  17. Focal Systems: Boosting Store Performance with an AI Retail Operating System and Real-Time DataFocal Systems details their implementation of an AI retail operating system using Confluent's managed Kafka and Flink. They describe how they ingest retail data (shelf images, product updates, inventory, sales), process it with computer vision and ML models on GCP, and use Flink SQL to compute real-time metrics and aggregates for shelf availability, out-of-stock events, and planogram compliance. The system outputs operational data to MongoDB and analytical data to Snowflake. The post highlights challenges with high-cardinality data, scalability, and the need for managed solutions, and explains how Confluent's platform addresses these by providing an event-driven streaming backbone and stream processing capabilities.
  18. Thunai Automates Customer Support with AI Agents and Data StreamingThunai leverages Confluent's data streaming platform to ingest and process real-time customer conversations (voice, chat, email, WhatsApp), transcripts, CRM updates, and user actions. This data feeds their 'Thunai Brain' knowledge base, enabling AI agents to understand context deeply and take safe actions across tools like CRM and to-do lists. The architecture prioritizes low latency, high throughput, and reliability to achieve high L1 task deflection rates and improve customer experience. They use Confluent for real-time ingestion, low-latency AI feedback, scalability, reliability, and governance, integrating with MongoDB Vector DB and LLMs like Gemini, Claude, and DeepSeek.
  19. Real-Time Context for Snowflake Intelligence with ConfluentThis post details how Confluent Intelligence, built on Kafka and Flink, delivers real-time context to Snowflake Cortex AI agents. It introduces Confluent's Streaming Agents and Real-Time Context Engine as solutions for context engineering, enabling AI systems to operate on current, trustworthy data. The post outlines three integration patterns: using Tableflow to convert Kafka topics to Iceberg tables in Snowflake, building event-driven Streaming Agents on Confluent Cloud for reasoning and action, and leveraging the Real-Time Context Engine to serve enriched data via MCP to Cortex agents. It highlights the architectural shift towards event-driven pipelines as the application itself and contrasts this with less effective approaches like batch pipelines or direct API calls.
  20. Detecting the Unexpected: Built-in Real-Time Anomaly Detection in Confluent Cloud for Apache Flink®Introduces built-in anomaly detection capabilities in Confluent Cloud for Apache Flink using the `ML_DETECT_ANOMALIES` function, which leverages ARIMA models. This function supports auto-ARIMA parameter determination, seasonal pattern detection via STL, adaptive training windows (`minTrainingSize`, `maxTrainingSize`, `updateInterval`), and confidence-based detection (`confidencePercentage`). The post details real-world use cases and provides an example SQL query for its implementation.
  21. How Superwall Scales with WarpStream and ClickHouse CloudThis post details Superwall's adoption of WarpStream and ClickHouse Cloud for their monetization platform. It highlights the evolution from a basic Kafka + single-instance ClickHouse setup to a managed, scalable solution. Key technical contributions include the use of WarpStream for Kafka-compatible streaming with reduced operational overhead, ClickHouse Cloud for elastic analytics with separation of storage and compute, and ClickPipes for managed ingestion from WarpStream to ClickHouse Cloud, ensuring durability and providing an error table for invalid records. The post emphasizes the benefits of this stack for handling high volumes of events (100 MB/sec) and large datasets (300+ TB) with near-real-time dashboard updates.
  22. Confluent Cloud Q4 2025: Data in Motion for AIIntroduces Phase 2 of Streaming Agents with Agent Definition (open preview) for simplified agent building and dynamic, iterative tool calling, and Observability and Debugging (open preview) for capturing agent interactions in an immutable log for visibility, auditability, and faster iteration. Also launches Real-Time Context Engine (early access) as a managed serving layer to materialize streaming data into a low-latency cache for AI applications, abstracting Kafka/Flink complexity and ensuring data consistency with automatic reprocessing. Python UDFs for Apache Flink are now in early access on AWS, expanding the language support for stream processing logic.
  23. Faster, Smarter, More Context-AwareStreaming AgentsIntroduces new capabilities for Streaming Agents including simplified agent definition with iterative tool calling, enhanced observability and debugging through structured immutable logs and Flink-powered recovery, and deeper integration with the Real-Time Context Engine for secure, reliable, and scalable delivery of live structured data to AI agents via MCP. Details new integrations with AWS, Google Cloud, Azure, various vector databases, and system integrators. Highlights use cases such as real-time fraud prevention, dynamic customer support, and intelligent supply chain optimization.
  24. Introducing Confluent Private CloudTableflow now offers GA for materializing Kafka topics as Delta Lake tables directly in customer-managed object storage (BYOS), with automatic table maintenance (compaction, vacuuming) and live metrics for observability. Unity Catalog Integration is also GA, automatically publishing Delta tables to Databricks Unity Catalog as external tables. Tableflow now supports upsert mode for CDC ingestion, handling billions of rows and terabytes of data with materialization lag under 15 minutes. Advanced error handling strategies (Suspend, Skip, Log) are introduced, with a Dead Letter Queue (DLQ) for logging, reprocessing, or analysis of malformed records. BYOK encryption is supported for both Confluent-managed and customer-managed object storage, with specific support for AWS KMS on AWS for Dedicated clusters.
  25. Introducing Real-Time Context Engine for AIIntroduces the Real-Time Context Engine, a new managed service in Confluent Cloud (Early Access) that acts as a serving layer for streaming data to AI systems. It materializes continuously processed and enriched enterprise data sets into a fast, in-memory cache and serves them via Model Context Protocol (MCP). This addresses the challenge of providing real-time, contextualized data to AI applications by unifying historical replay, continuous processing, and real-time serving. The engine abstracts Kafka and Flink complexity, offering built-in authentication, RBAC, and audit logging, and automatically reprocesses data when upstream definitions change.
  26. Confluent + Google Cloud: Real-Time Backbone for Agentic AIThis post introduces the MCP/A2A proxy, a new component that acts as an intermediary for AI agents to communicate with each other (A2A) and with external resources (MCP). The proxy translates disparate MCP and A2A interactions into standardized messages within Confluent Cloud, providing a unified registry, protocol transformation, and service exposure capabilities. It enables existing microservices, legacy applications, and Flink jobs to be consumable by AI agents. The post also emphasizes how Confluent's event-driven architecture and asynchronous communication patterns, leveraging Kafka topics, support decoupled agent workflows and reliable message delivery. It highlights Confluent's role in providing real-time data to LLMs and knowledge bases, and acting as a proxy for inter-agent communication and workflow orchestration.
  27. Better-Governed Data Lake Architectures with Confluent TableflowThis post introduces Confluent Tableflow as a solution to data lake governance challenges, focusing on data quality, error handling, and discoverability. It advocates for a shift-left analytics approach, emphasizing early governance and metadata management. Tableflow materializes Kafka topics and schemas into open table formats (Iceberg, Delta Lake), integrating with data warehouses and analytics engines. It leverages Schema Registry for structured data representation and schema evolution, and provides direct catalog synchronization with platforms like AWS Glue, Polaris, and Databricks Unity, reducing reliance on batch jobs for metadata updates.
  28. How Real-Time Streaming Prevents Fraud in Banking & PaymentsThis post details how real-time data streaming, using technologies like Apache Kafka and Apache Flink, fundamentally changes fraud detection in banking and payments by enabling immediate anomaly detection, transaction scoring, composite event identification, and automated real-time responses. It contrasts this with traditional batch processing, highlighting reduced latency and financial losses. The post also emphasizes the importance of governance, data accuracy, and data lineage for trust and compliance in financial-grade alerting systems. Case studies like EVO Banco's 99% reduction in fraud losses are cited as evidence of the impact.
  29. How to Build Real-Time, Event-Driven Automation for Business InsightsThis post details how to build real-time, event-driven automation for business insights using event-driven workflows and AI agents. It outlines the architectural components, including event brokers (Apache Kafka) and processing layers (Apache Flink/Kafka Streams), and provides best practices for implementation such as using Schema Registry, testing under real load, and planning for anomalies. It also presents real-world use cases like fraud detection, customer churn prevention, and inventory optimization, emphasizing the benefits of immediate responsiveness, scalability, and personalization.
  30. Confluent Cloud Q3 '25 Unlocks Cost-Effective Data StreamingIntroduces Streaming Agents in Open Preview, enabling the building, deployment, and orchestration of event-driven AI agents directly within stream processing pipelines on Apache Kafka and Apache Flink. Key features in preview include tool calling with MCP, Connections for secure and reusable integrations with models and databases, and external tables/search for data enrichment. GA features include AI model inference and embeddings within Flink SQL, and built-in ML functions for time-series forecasting and anomaly detection. Also introduces Custom Single Message Transforms (SMTs) for fully managed connectors, allowing users to upload and manage their own transformation logic. Expands Tableflow functionality with Delta Lake and Databricks Unity Catalog integration, including automated schema evolution and table maintenance, and upsert materialization for CDC. Cluster Linking is now available on Google Cloud.
  31. Introducing Streaming Agents on Confluent CloudIntroduces Streaming Agents on Confluent Cloud, enabling event-driven AI agents natively on Apache Flink. This feature unifies stream processing and AI workflows, allowing agents to monitor and act on real-time business events. Key technical contributions include: event-driven architecture with Flink and Kafka for asynchronous processing and replayability; seamless integration with LLMs, tools, and data systems via Model Context Protocol (MCP) and native connectors; real-time context access for improved decisioning; built-in ML functions for forecasting and anomaly detection; secure and governed event flows with replayability for auditing and debugging; and enhanced external tables and vector search for data enrichment. The post details how Streaming Agents leverage Flink APIs for development and deployment, enabling engineers to build context-aware automation.
  32. Modernize Legacy Data for Real-Time AI with Oracle & MongoDBThis post details building a real-time personalization engine by ingesting data from Oracle databases to Kafka using the Oracle XStream CDC Source Connector, enriching it with Flink and LLMs (GPT-4 for descriptions, OpenAI for embeddings), and sinking embeddings to MongoDB for semantic search. It demonstrates how to use Kafka and Flink for real-time data pipelines to power AI applications, specifically for product recommendations based on semantic similarity. The post includes Flink SQL examples for model creation and data processing.
  33. Zero-ETL vs Tableflow | Integrating Apache Kafka® and Zero-ETL via Confluent TableflowThis post introduces Confluent Tableflow as a solution that materializes Kafka topics into open table formats (Iceberg, Delta Lake), effectively bridging operational and analytical estates. It contrasts Tableflow with traditional ETL and zero-ETL approaches, highlighting Tableflow's advantages in reusability, loose coupling between producers and consumers, and cost-effectiveness. The post details how Tableflow leverages Kafka's log abstraction and open table formats to provide a unified system for data access, enabling stream processing and table-oriented access simultaneously.
  34. Transforming Confluent Operations With GenAIThis post details the integration of NeuBird's Hawkeye, a GenAI-powered SRE assistant, with Confluent Cloud's observability stack. It describes how Hawkeye automates incident investigation and resolution by correlating Prometheus metrics, Confluent Cloud metrics, producer logs, and CloudWatch audit logs. The technical architecture includes Kubernetes deployment (Amazon EKS), Prometheus Alertmanager integration for incident triggering, and secure, read-only access for Hawkeye to telemetry sources. A specific scenario of authorization revocation is used to illustrate the traditional manual troubleshooting workflow versus the automated Hawkeye workflow, highlighting reduced MTTR.
  35. Unlocking Real-Time Analytics With Tableflow & AWS Glue CatalogThis post announces the General Availability (GA) of Tableflow, a feature that materializes Kafka topics into Apache Iceberg tables. It details the technical capabilities of Tableflow, including data conversion from Avro, JSON, or Protobuf to Iceberg-compatible schemas and Parquet files using Schema Registry, automatic schema evolution detection, catalog syncing with AWS Glue Data Catalog, Snowflake Open Catalog, and Databricks Unity Catalog, and automatic table maintenance like compaction and snapshot expiration. The post also elaborates on the integration with AWS Glue Data Catalog, explaining how Tableflow publishes Iceberg table metadata pointers to enable AWS Analytics services (Athena, Redshift Spectrum, EMR) and Amazon SageMaker Lakehouse to access these tables. A walk-through demo of the integration process is provided.
  36. Why Flink Agents Are the Future of Enterprise AIIntroduces Flink Agents, a new Flink sub-project developed collaboratively with Alibaba, to bridge the gap in building enterprise AI agents. Key contributions include: 1. First-class agent semantics within Flink APIs for model inference, tool invocation, and contextual search. 2. Dynamic topology support for complex reasoning patterns like ReAct workflows, moving beyond traditional sequential data processing. 3. Enhanced observability for agent state, tool invocations, model inference calls, and decision traces. 4. Native Model Context Protocol (MCP) support for seamless integration with MCP-compatible tools and services. The post also highlights the developer experience with Flink's Table API for agent creation and application to data streams, emphasizing unified infrastructure, end-to-end consistency, fault tolerance, and replayability.
  37. Build an AI Personalization Engine with Confluent & DatabricksThis post introduces a quickstart for building an AI-powered marketing personalization engine by integrating Confluent Cloud and Databricks. It details the architecture and data flow, highlighting the use of Confluent's Oracle XStream CDC Source Connector for ingesting operational data, Apache Flink for stream processing and AI model inference, and Tableflow for seamless synchronization of Kafka data as Delta Lake tables in Databricks. The quickstart guides users through setting up the environment, deploying cloud infrastructure, ingesting mock data, processing and enriching data with Flink, streaming data products to Delta Lake with Tableflow, and generating insights and campaigns with Databricks.
  38. Delta Lake & Unity Catalog Now Integrate With TableflowThis post announces major features for Tableflow, enhancing its integration with Databricks. Key updates include support for Delta Lake (Open Preview) and robust integration with Databricks Unity Catalog (Open Preview expected by end of June). Tableflow now automatically materializes Kafka topics into Delta Lake tables, supporting Avro, JSON, or Protobuf formats and handling schema evolution. It also syncs tables as external tables in Databricks Unity Catalog, providing end-to-end lineage and centralized governance. The process involves configuring an S3 bucket for storage and integrating with Databricks Unity Catalog via a provider integration, allowing Databricks users to query these tables directly.
  39. Inside Tableflow GA: Real-Time Kafka to IcebergThis post announces the General Availability (GA) of Tableflow, a product that materializes Apache Kafka topics into Apache Iceberg tables. It details the technical architecture of Tableflow, which leverages Kora's storage layer to convert Kafka segments into Parquet files. A new metadata publishing service uses Schema Registry to generate Iceberg metadata transaction logs, handling schema mapping, evolution, and type conversions. Key features highlighted include data conversion (Avro, JSON, Protobuf to Iceberg/Parquet), automatic schema evolution, catalog syncing (AWS Glue, Snowflake Open Catalog, Apache Polaris, Unity Catalog), and table maintenance (compaction, snapshot expiration). The post also explains Snowflake Open Catalog's role in managing Iceberg tables and provides a walk-through of the integration process, emphasizing the use of AWS AssumeRole for provider integration and external volumes/catalog integrations within Snowflake to access the Iceberg data stored in S3.
  40. The Future of AI Agents Is Event-Driven | ConfluentThis post argues that the future of AI agents lies in event-driven architectures (EDA) powered by data streams, rather than solely focusing on model improvements. It highlights the challenges of scaling agents, which are fundamentally distributed systems problems requiring loose coupling for data and tool integration. The article contrasts tightly coupled RPC/API approaches with EDA, emphasizing how asynchronous event-driven communication enables agents to share information, act in real-time, and integrate with broader systems without rigid dependencies. It draws parallels to the scaling challenges faced by early social networks and positions EDA as the foundation for resilient and scalable agentic systems, enabling agents to act as microservices with informational dependencies.
  41. Building Real-Time Multi-Agent AI With ConfluentThis post details how Agent Taskflow, a multi-agent orchestration platform, leverages Confluent's data streaming platform (Kafka, connectors, Stream Governance) to build real-time, observable, and fault-tolerant communication for AI agents. It highlights the technical challenges of multi-agent communication, observability, fault tolerance, scalability, and identity/permissioning, and explains how Confluent addresses these through an event-first architecture. Specific integrations include PostgreSQL Sink Connector for logs, Apache Iceberg Sink Connector for historical data, and a custom webhook source connector. Schema Registry is used for strict compatibility and data quality, and Stream Lineage for debugging. RAG is implemented by vectorizing events and storing them in Qdrant for agent context.
  42. Secure Inter-Organizational Data SharingThis post details how Confluent's data streaming platform enables secure, reliable, and scalable inter-organizational data sharing. It highlights the strategic value and technical challenges of cross-boundary data exchange, contrasting traditional methods with data streaming. Three Confluent methods are presented: Stream Sharing for simple topic sharing with external parties, Cluster Linking for replicating data between clusters (including across organizations), and using Apache Kafka or REST API client applications. Stream Sharing emphasizes authenticated access, encryption, and schema enforcement for providers, and frictionless consumption for recipients. Cluster Linking is presented for more complex replication scenarios, often involving a gateway cluster. The post also touches on the importance of data quality and schema compatibility for recipients.
  43. The AI Silo Problem: How Data Streaming Can Unify Enterprise AI AgentsThis post introduces the concept of an 'AI Silo Problem' where disparate AI agents operate in isolation, hindering enterprise-wide intelligence and automation. It proposes a solution using a data streaming platform (like Kafka) as an event-driven backbone for inter-agent communication. Key components include an agent registry for structured governance, dedicated Kafka topics for agent event exchange, and stream processing (e.g., Flink) with LLM-based event mapping for intelligent routing and orchestration across heterogeneous agent platforms. This enables agents to communicate, share insights, and coordinate actions in real-time, overcoming the limitations of static mappings in controlled environments.
  44. Confluent Cloud Q1 ’25 introduces Tableflow, Freight clusters, and Flink AI enhancementsThis post announces the general availability of Tableflow, which represents Kafka topics and schemas as Apache Iceberg or Delta Lake tables for data lakes and warehouses. It also introduces Freight clusters, a cost-effective option for high-throughput, relaxed-latency workloads that write directly to object storage (e.g., S3) for up to 90% cost savings. Enterprise clusters now support Bring Your Own Key (BYOK) on AWS and are increasing their maximum capacity to 32 eCKUs. Apache Flink® enhancements include Flink Native Inference for open source and fine-tuned models, Flink Search for vector database queries (MongoDB, Elasticsearch, Pinecone) to support RAG, and built-in ML functions for time-series forecasting and anomaly detection. The Confluent for VS Code extension is now generally available. The Oracle XStream CDC Premium Connector is also now available.
  45. Tableflow is now generally availableThis post announces the general availability of Tableflow, a new product that allows users to materialize Apache Kafka topics as Apache Iceberg™ or Delta Lake tables. It details the challenges of traditional Kafka-to-data-lake pipelines, highlighting fragility, cost, and complexity. Tableflow solves these issues by automatically converting Kafka segments and schemas (Avro, JSON, Protobuf) into Iceberg/Delta-compatible schemas and Parquet files, leveraging Confluent's Schema Registry. It handles schema evolution, catalog syncing with partners (AWS Glue, Databricks Unity Catalog, Snowflake Open Catalog), and table maintenance like compaction and snapshot expiration. The post also emphasizes the synergy with Apache Flink® for real-time data transformation before landing in tables, and lists various partner integrations and analytical tools that support Tableflow's output.
  46. Real-Time Toxicity Detection in Games: Balancing Moderation and Player ExperienceThis post details a real-time toxicity detection system for gaming. It uses Confluent Cloud for Kafka transport and Apache Flink for initial triage with lightweight ML models. High-risk messages are sent to Databricks for deeper NLP analysis. Tableflow is used to convert Kafka streams to Delta tables for Databricks. The system distinguishes between 'OK', 'Toxic', and 'Requires NLP Determination' messages, with a feedback loop for continuous model training using human moderator input. The architecture prioritizes low latency for the majority of chats while ensuring accurate detection of nuanced toxicity.
  47. Your GenAI Project Needs a Data Streaming PlatformThis post argues that traditional data architectures (silos, batch processing, ELT/reverse ETL) create a 'data liberation problem' that hinders Generative AI projects by providing stale or inaccessible data. It advocates for data streaming platforms as the solution, enabling real-time data flow, decoupling systems, making unstructured data AI-ready, improving data quality, and unifying ecosystems. The core technical argument is that streaming platforms allow data processing closer to the source, ensuring data is fresh and actionable for AI applications.
  48. Building AI Agents and Copilots with Confluent, Airy, and Apache FlinkThis post details how Airy leverages Confluent Data Streaming Platform, Apache Flink, and AI Model Inference to build AI agents and copilots. It highlights the use of Confluent connectors for data ingestion, Flink for stream processing and data transformation, and AI Model Inference for creating embeddings and RAG. The post also mentions the use of Stream Governance for data contracts and schema management, and Tableflow for integrating with Iceberg. Key challenges addressed include data governance, accuracy of GenAI outputs, and scalable infrastructure.
  49. How Confluent and Databricks Are Unlocking Real-Time AIThis post details an expanded partnership between Confluent and Databricks focused on simplifying the integration between analytical and operational systems to accelerate AI and real-time data application deployment. The core technical contribution is a Delta Lake-first integration that enables bidirectional data flow between Confluent's Tableflow and Databricks' Unity Catalog. This integration allows for transforming Kafka event streams into Delta Lake tables for AI model training and serving (leveraging Databricks' Mosaic AI), and feeding AI-generated insights back into Kafka via updated Delta tables. It emphasizes unified governance and lineage tracking through the integration of Databricks Unity Catalog and Confluent Stream Governance, aiming to eliminate data silos and enable responsible AI deployment at scale.
  50. Allium and Confluent: How to Build a Foundational Data Platform for BlockchainThis post details how Allium uses Confluent's data streaming platform to ingest data from blockchain nodes, transform it using Google Dataflow, and then stream it to Snowflake and a self-built observability service via Confluent's managed connectors. It also highlights the use of Confluent Cloud's Stream Sharing for previewing data with prospects. The future vision includes leveraging Confluent Cloud for Apache Flink for upstream processing and Tableflow to create Iceberg tables.
  51. Optimizing Supply Chains with Data Streaming and Generative AIThis post details a specific implementation of real-time enterprise RAG within a supply chain context. It describes a system where shop assistants use a GenAI-enabled mobile app to photograph products. This image data is streamed via Confluent to Azure AI's Vision service for scanning and classification. The output is then processed by Apache Flink, joined with real-time inventory data from SQL Server, converted into vector embeddings, and stored in Azure Cosmos DB. This enables retrieval-augmented generation (RAG) by an LLM (OpenAI) to provide actionable instructions to shop assistants, such as 'Keep', 'Remove', or 'Discount', thereby reducing waste and boosting sales. The architecture highlights the integration of computer vision, stream processing, vector databases, and LLMs for real-time decision-making.
  52. Reworkd Scales Real-Time Web Scraping With Data Streaming and GenAIReworkd utilizes Confluent's Data Streaming Platform to scale real-time web scraping for GenAI applications. They employ agentic AI (GPT-4) to automate the generation and validation of web scraping code, using RAG to create embeddings for context understanding. Scraped data is streamed into Kafka topics for validation, transformation, and deduplication. Schema Registry is used for data governance, and data is then sent to MySQL databases and ClickHouse via Kafka Connect and Debezium. The processed data is exposed via a REST API for customer consumption. This approach significantly accelerates data extraction, reduces costs, and ensures data accuracy for AI models.
  53. Materialize Your Topics with Tableflow and Amazon SageMaker LakehouseThis post announces the upcoming open preview of Confluent Tableflow, highlighting its integration with Amazon SageMaker Lakehouse. It details how Tableflow materializes Kafka topics into Apache Iceberg tables, leveraging AWS Glue Data Catalog for metadata management. The post explains the benefits of Apache Iceberg for data lakes and how Tableflow simplifies the process of feeding operational Kafka data into analytical estates, reducing the complexity of custom data pipelines. It provides a walkthrough of a use case involving CDC streams from PostgreSQL to Kafka, then to Iceberg tables accessible by Amazon Athena, Redshift, and EMR.
  54. Headless Data Architecture, Part 2: How to Build ItThis post elaborates on the headless data architecture concept, building upon the previous article. It details the problems with traditional multi-hop and medallion architectures (slowness, cost, brittleness, data sprawl, trust issues) and proposes a shift-left approach. It defines a 'data product' as a combination of a Kafka stream and an Iceberg table, accessible through various 'processing heads'. The post outlines strategies for building this architecture, including creating source-to-stream workflows (using CDC or direct production) and materializing streams into Iceberg tables (via Kafka Connect or services like Tableflow). It also provides a modular, incremental approach to adopting this architecture by selecting commonly used data sets, identifying sources, creating parallel workflows, and validating the new Iceberg tables before migrating analytical jobs.
  55. Headless Data Architecture, Explained for DevelopersThis post introduces and explains the concept of a 'headless data architecture,' which decouples data storage, management, and access from the services that interact with it. It details how Apache Kafka serves as the streaming layer and Apache Iceberg, with its table storage, catalog, transaction, and time-travel capabilities, serves as the table layer. The post emphasizes the benefits of this architecture, including reduced data duplication, simplified coordination, and the flexibility to choose processing engines. It also contrasts this approach with traditional data lake architectures, highlighting its modularity and pluggability. The post mentions connectors like Kafka Connect and managed services like Tableflow for materializing streams into Iceberg tables.
  56. Revolutionizing Computational Chemistry with Data StreamingAtomic Tessellator migrated from a batch-based ETL system using RabbitMQ, Celery, and Django to Confluent and Apache Kafka for their computational chemistry platform. This migration addressed issues of slow, siloed, and fragile data infrastructure. The new architecture leverages Kafka to ingest complex molecular data from databases and generative models, streaming it to an evaluation application for replicating chemical reactions. The system allows for dynamic scaling of pipelines to meet experimental demands, real-time adjustment of pipeline components (e.g., 'entropy injection rate'), and provides real-time visibility into pipeline status via JMX metrics. They are also exploring Apache Flink for building 'pipeline templates' and integrating LLMs for automated experiment design, execution, and debugging.
  57. Let Flink Cook: Mastering Real-Time Retrieval-Augmented Generation (RAG) with FlinkThis post details how to build a real-time Retrieval-Augmented Generation (RAG) pipeline using Apache Kafka and Apache Flink in Confluent Cloud. It introduces new Flink SQL functions like `ml_predict()` and `federated_search()` to integrate LLMs and vector stores directly into data processing pipelines. The post outlines two key patterns: data preparation (chunking, vector embedding creation using `ml_predict()` with OpenAI embeddings) and inference (retrieving context and augmenting prompts). It highlights the benefits of using Flink for real-time data orchestration, state management for LLMs, and post-processing for reliability checks. The contribution is the practical application of Flink SQL for RAG, enabling real-time data grounding for LLMs.
  58. Busie: Building a Charter Platform From the Ground Up with Confluent CloudBusie built a charter platform on Confluent Cloud, migrating from a monolithic and RESTful microservices architecture to an event-driven one. They integrated data from 13 PostgreSQL and MongoDB databases using Confluent Cloud's managed connectors and self-managed connectors on AWS ECS with Fargate. They developed a 'Charter Experience Finder' (ChEF) search engine that aggregates and enriches data from disparate sources within Confluent Cloud to create a unified data model optimized for search, pagination, and filtering. This shift-left processing approach reduced the loading time of their Range Operator Portal's core view by nearly 300%. They are also exploring the use of fully-managed Apache Flink for real-time analytics dashboards, yield-based pricing, and real-time vehicle location reporting.
  59. BigQuery Continuous Queries & Confluent Yield Real-Time AnalyticsThis post details the integration of BigQuery continuous queries with Confluent Cloud. It explains how BigQuery continuous queries can export results to Pub/Sub, which is then consumed by Confluent Cloud's Google Cloud Pub/Sub Source Connector. The post provides a step-by-step guide for setting up this integration, including configuring BigQuery continuous queries, setting up Pub/Sub topics and subscriptions, and configuring the Confluent Cloud Pub/Sub Source Connector. It highlights the benefits of this joint solution for real-time data-driven decisions, enhanced application development, streamlined data pipelines, improved data governance, cost optimization, and unlocking new business opportunities.
  60. Using Kafka-Powered AI Models to Predict and Prevent Sepsis at City of HopeThis post details how City of Hope implemented a real-time data streaming infrastructure using Apache Kafka to power an AI model for predicting and preventing sepsis in bone marrow transplant patients. The architecture involves ingesting HL7 messages from Epic via Corepoint, converting them to JSON, and streaming them into Kafka topics (vitals, lab results, medication orders, ADT, substance administration). Microservices process this data, trigger the predictive model, and feed predictions back into Epic to alert clinicians. Confluent Cloud on Azure is used for scalability and reliability, with ksqlDB used for data inspection. The system leverages Kafka's immutable log for state retrieval if the model crashes.
  61. Enhanced Cybersecurity with Real-Time Log Aggregation and Analysis | ConfluentThis post details the implementation of a real-time log aggregation and analysis system for cybersecurity using Confluent Cloud. It outlines the challenges of managing large volumes of heterogeneous security logs and the benefits of using a data streaming platform for real-time detection of malicious activity. The architecture involves ingesting various log types (login, firewall, IPS, web) into Confluent Cloud topics, using Apache Flink for stream processing to enrich data and create security-focused data products (e.g., SuspiciousLogins, custom alerts by joining IDS and Logins streams), and then sinking processed data to Elasticsearch for Kibana dashboards and Google BigQuery for AI/ML modeling. Sample Flink SQL queries for detecting multiple login attempts from the same IP and identifying login attempts from known attack IPs are provided. The post emphasizes the 'shift-left' approach with stream processing for faster threat detection and response.
  62. Build a Scalable and Up-to-Date Generative AI Chatbot with Amazon Bedrock and Confluent CloudThis post details the architecture and implementation of a generative AI chatbot using Confluent Cloud and Amazon Bedrock. It highlights the use of Kafka for real-time data streaming to keep the AI agent updated, a microservices architecture for agility and scalability, and Amazon Bedrock with Anthropic's model for conversational AI. The chatbot processing microservice utilizes KStream API and KCache for real-time responses and dynamic cache updates. The pre-approval microservice leverages Amazon Bedrock to extract mandatory information from conversations into JSON format for an automated approval process.
  63. Next-Gen Customer Loyalty Programs with Data Streaming | ConfluentThis post details how Confluent's data streaming platform can be used to build next-generation customer loyalty programs. It highlights the use of Confluent Cloud to ingest data from various sources (CRM, databases, marketing systems) using pre-built connectors. It emphasizes stream processing with Apache Flink to join, enrich, and transform data in real-time, enabling personalized rewards and dynamic pricing. A key contribution is the demonstration of Flink's AI Model Inference feature to call external AI engines for predictive analytics and personalized recommendations, exemplified by a Flink SQL query to calculate customer loyalty levels.
  64. Data Streaming in Healthcare: Achieving the Single Patient ViewThis post details how Confluent's data streaming platform, including Apache Flink, can be used to build a Single Patient View (SPV) in healthcare, enabling real-time experimental treatment-matching services. It highlights the challenges of interoperability, regulatory compliance, scalability, observability, and disaster recovery in healthcare data integration and demonstrates how a managed, serverless data streaming platform addresses these issues, including a case study of a healthcare organization using Confluent Cloud for a treatment-matching service.
  65. Making Predictive Customer Support a Reality for TelcosThis post details how telcos can leverage data streaming for predictive customer support by ingesting and processing real-time data (customer behavior, network performance, subnetwork data) to train predictive models that preemptively detect and rectify anomalies. It outlines the challenges of siloed data, high data volume, disconnected teams, batch ETL, legacy technologies, and lack of scalability. Confluent's platform enables streaming data on-premises and across clouds, connecting disparate data sources with pre-built connectors, processing streams with Apache Flink to build live data products, and governing data with Stream Governance (Stream Lineage, client-side field-level encryption, RBAC, Schema Registry, Data Portal). The solution implementation includes a hybrid architecture with connectors for IBM MQ, Splunk, PostgreSQL, and SQL Server, mirroring topics to Confluent Cloud for stream processing and data product creation, and downstream enrichment for ML model training and dashboards. A Flink SQL query example is provided for identifying unhealthy network situations based on call success rates and clearing codes.
  66. Building Event-Driven GenAI Applications in 4 StepsThis post outlines a four-step methodology for building event-driven Generative AI applications, focusing on data augmentation, inference, workflows, and post-processing. It emphasizes the use of Apache Kafka and data streaming platforms to integrate disparate data sources, create embeddings for vector stores, and decouple LLM inference from user-facing applications. The approach leverages event-driven patterns to manage complexity, enable independent scaling of teams and services, and treat AI components as modular, substitutable parts. Specific technical aspects discussed include using Kafka consumers to pull data from connectors, process it, and create embeddings for vector stores, and treating LLM calls as asynchronous operations to avoid blocking web application threads. The post also touches upon using LangChain for workflow composition and the importance of post-processing for validating LLM outputs.
  67. Real-Time Order Management: The Key to Streamlining Your Business OperationsThis post details how to leverage Confluent for real-time order management, focusing on data accuracy, order authenticity, and fraud detection using stream processing and connectors. It highlights the use of Flink for stream processing and provides a streaming architecture diagram and ksqlDB/Flink queries for enriching order data, validating against inventory, and sending notifications. The business impact includes reimagined customer excellence, ensured order integrity through Stream Governance, and real-time inventory replenishment.
  68. Modern data streaming platforms bring better customer travel experiencesThis post details how cruise lines are leveraging Confluent Cloud and Apache Kafka to build real-time customer experience platforms. It describes the challenges of legacy systems, limited bandwidth on ships, and the need for seamless ship-to-shore data replication. The technical solution involves using CDC-based connectors to capture data changes, logging events into Kafka topics, and employing Kafka Streams for event-driven microservices. Confluent for Kubernetes is used for managing onboard clusters, while Confluent Cloud is utilized onshore for managed connectors and stream processing. Cluster Linking is mentioned for ship-to-shore replication. The post highlights the benefits of real-time data streaming for improving customer experiences in the travel industry.
  69. Delivering Real-Time Manufacturing Predictive MaintenanceThis post details the application of Confluent Cloud and Kafka in the manufacturing industry for real-time visibility during engine production. It explains how manufacturing data (e.g., machining tolerances, test results) is captured and streamed to Confluent Cloud. The post outlines the use of stream processing (Kafka Streams, ksqlDB, Flink) for real-time analysis of engine and tooling quality, and for training ML models for predictive maintenance. It emphasizes the benefits of real-time data for reducing scrap, rework, and unplanned downtime, and for extending equipment lifespan. The post also mentions the use of 120+ pre-built connectors for integrating with various data sources and downstream systems.
  70. Data Portal for Confluent Stream GovernanceIntroduces Data Portal, a self-service UI for discovering, exploring, and accessing Kafka topics on Confluent Cloud. It leverages Stream Catalog and Stream Lineage for data discovery and understanding. Features include searching topics by metadata, requesting access via an approval workflow, and querying data with Flink SQL. This enhances the user experience for data users interacting with Kafka data streams.
  71. Data Streaming and AI are Better When They’re TogetherThis post emphasizes the synergy between data streaming and AI, positioning data streaming platforms as the 'central nervous system' for AI/ML models. It details how Confluent Cloud simplifies AI/ML integration by enabling model embedding within event streams, facilitating real-time feature computation, online inferencing, and continual learning. The post highlights Confluent Cloud's capabilities in abstracting complexity, supporting edge deployment via librdkafka, and integrating with mobile applications. It also mentions the use of Apache Flink for high-performance stream processing and fault tolerance, and how Confluent Cloud lowers operational barriers for AI/ML engineers.
  72. Real-Time Risk Analysis with Stream ProcessingThis post introduces the capability of providing real-time risk analysis by leveraging data streaming. It details how to ingest data from various sources continuously, throughout the day, including portfolio positions, trading activity, market data, and news feeds, immediately as something happens. The core problem addressed is enabling rapid data access for risk managers to proactively assess uncertainties and take decisive action to effectively manage exposures, moving away from slow, batch-based risk analysis.
  73. Secure Data Exchange for Financial InstitutionsThis post introduces the application of Confluent Cloud for secure data exchange in the financial services industry (FSI). It details how Confluent Cloud clusters at the edge can be used to overcome challenges in regulatory reporting, trading services, portfolio management, investment research, and investment servicing. Key features highlighted include Streaming Governance for standardized data formats and data contracts, data encryption (BYOK, CSFLE) for data privacy, OAuth and RBAC for security and onboarding, Client Quotas for multi-tenancy, and Observability for chargebacks and monitoring. The post emphasizes the benefits of using Confluent Cloud for seamless, secure, and timely data exchange, including improved efficiency, scalability, and fault tolerance compared to traditional methods.
  74. Real-Time Business Analytics Using AIThis post introduces the challenges and solutions for real-time business analytics using AI/ML, focusing on data quality and scalability. It highlights Confluent Cloud's capabilities in addressing these challenges through managed Kafka Connectors for data completeness, ksqlDB and Apache Flink for data context and transformation (including OpenAI API integration), and Schema Registry and Stream Governance for data consistency. The post also touches upon infrastructure scaling and continuous model training, positioning Confluent Cloud as a comprehensive platform for AI/ML-driven analytics.
  75. Generative AI: An Introduction to GPT, LLMs, and MoreThis post introduces the foundational concepts of Generative AI, Large Language Models (LLMs), and GPT, explaining their underlying technologies like transformers and the pre-training process. It highlights the challenges in building production-grade GenAI applications, such as context, memory, private data handling, and scaling. The post proposes the Kappa Architecture and Confluent's Kora engine as solutions to these challenges, setting the stage for a series that will delve into building a GenAI application with Retrieval-Augmented Generation (RAG). It also introduces prompt engineering as a key skill for interacting with LLMs.
  76. Real-Time Demand Forecasting with ConfluentThis post details how Acme Inc. uses Confluent Cloud for real-time demand forecasting by integrating event data from various sources into a continuous data pipeline. The architecture leverages Confluent Cloud as a central hub for data ingestion and distribution, feeding data into Elasticsearch, MongoDB, and Snowflake via Snowpipe. The primary technical contribution is the architectural pattern of using Confluent Cloud to aggregate, enrich, and standardize event data for downstream analysis and forecasting, addressing challenges of scalability, reliability, and simplified operations compared to open-source Kafka.
  77. In-store Personalization with Confluent Data StreamingThis post details how to build an in-store personalization system using Confluent Data Streaming. It outlines the requirements for such a system, including high availability, scalability, and low latency. The implementation example focuses on using a mobile app with geofencing and a notification platform (Ably) to trigger real-time offers and information based on customer location and behavior within the store. It also discusses leveraging customer data platforms, streaming data from databases via CDC, and using ksqlDB for real-time insights and ML model integration.
  78. The Role of Real-Time Interoperability to Healthcare PayersThis post details how real-time data streaming with Confluent can be used to address technical challenges in healthcare payer interoperability, specifically by overcoming data silos and legacy communication methods. It outlines an example architecture using Kafka Produce-Consume and CDC patterns to enable real-time data availability for downstream systems and microservices, ultimately improving care coordination, efficiency, and member experience.
  79. AI Data Streaming: Building AI, ML, and LLM Apps Just Got EasierThis post announces Confluent's 'Data Streaming for AI' initiative, expanding its ecosystem to include critical AI technologies and committing to product enhancements for building AI apps faster. It highlights the challenges of integrating AI models with real-time, trustworthy data, positioning data streaming as the foundation for the modern AI stack. Key technical contributions include plans for fully managed connectors for the AI stack, GenAI API calls within Apache Flink SQL (preview), and an AI Assistant for Confluent users. It also details how Confluent's platform enables dynamic, real-time knowledge repositories, real-time context integration at query execution, agile experimentation, and governed, secure AI data. Partnerships with vector databases for RAG and cloud providers (AWS, Google Cloud, Microsoft Azure) for AI integrations are also emphasized.
  80. Achieving Same Day Delivery with Confluent Data StreamingThis post details a grocery delivery company's migration from a self-managed Apache Kafka on Azure HDInsight to Confluent Cloud to enable same-day delivery. The previous setup suffered from maintenance burdens, missed upgrades/patches, frequent pipeline failures, lack of managed connectors, and no stream processing. The migration to Confluent Cloud leverages multi-AZ dedicated clusters with VNET peering for high availability, pre-built managed connectors (Debezium MongoDB, SQL Server, JDBC Sink) for data ingestion and egress, and Schema Registry for data consistency. The .NET client library was used for application integration. The result is improved reliability, reduced downtime, and freed-up engineering resources for innovation.
  81. Building Event-Driven Architectures: Top 5 Best PracticesThis post introduces two patterns for integrating Confluent Cloud with AWS Lambda to build event-driven applications: the fully managed AWS Lambda Sink Connector and native event source mapping (ESM). It details the benefits and considerations of each pattern, including throughput, latency, scaling, and error handling. Additionally, it provides five best practices for building event-driven architectures with Confluent and Lambda: batching calls to Lambda, implementing robust error handling in Lambda code to avoid duplicates and stalled partitions, using Lambda functions idempotently, and keeping long-lived connections to Kafka outside the handler for better efficiency.
  82. Real-time recommendations for mobile phones using data streamingThis post details the implementation of a real-time recommendation system for mobile phones using Confluent Cloud. It addresses the challenges of scaling personalization for millions of users and the limitations of traditional approaches. The architecture leverages Confluent Cloud for aggregating content, serving personalized recommendations, and tracking user interactions. Key technical contributions include using CDC source connectors to stream user data from Postgres to Kafka, creating ksqlDB materialized views for real-time user enrichment, and utilizing fanout for efficient data consumption by recommendation and reporting services. It also highlights the use of Stream Lineage for debugging data flow issues and improving operational visibility.
  83. Real-Time AI: Live Recommendations Using Confluent and RocksetThis post details how Confluent and Rockset enable real-time AI applications by providing a robust architecture for contextualizing AI models with fresh data. It explains two patterns for contextualizing models: extending models with real-time data and feeding models with real-time data. The post highlights the challenges of traditional batch-oriented data pipelines and introduces Confluent Cloud as a data streaming platform and Rockset as a real-time analytics database with vector search capabilities. It uses the example of Whatnot, an e-commerce innovator, to illustrate how this pattern is used for real-time recommendations based on user activity and livestream data. Key technical aspects include low data latency, high query latency, efficient upserts, metadata filtering alongside vector search, and compute-compute separation for scaling.
  84. Maximizing the Power of AI with Data StreamingThis post details how data streaming, powered by Apache Kafka, fuels AI by enabling continuous training on data streams, efficient data synchronization to ML platforms, real-time application of AI models, and high-volume data processing. It emphasizes the creation of a real-time data mesh to democratize data accessibility and reduce integration complexity. The post also highlights the importance of data governance and security, mentioning RBAC and ABAC for secure and scalable data access to AI systems.
  85. How Mobile Premier League Reduced Player Churn with Confluent CloudThis post details how Mobile Premier League (MPL) migrated from a batch pipeline system to Confluent Cloud to reduce player churn. MPL implemented real-time streaming pipelines to address poor matchmaking, delayed decision-making, and fraud issues. They leveraged Confluent Cloud to ingest and process events in real-time, enabling optimized matchmaking with personalized offers, customized onboarding journeys for new players, and real-time fraud detection and prevention. The post highlights the operational efficiency and cost savings achieved by using a managed Kafka service.
  86. Automatic Detect PII in Real Time with Machine LearningThis post introduces a PII Detection stream processing app, PII UDFs/UDTFs for ksqlDB, and a PII SMT for Kafka Connect. These solutions leverage NLP and pattern recognition models to identify and redact Personally Identifiable Information (PII) in real-time within data streams. The PII Detection app acts as a pass-through filter, inspecting messages for PII entities and redacting them while retaining other data, and publishing entity metadata events for alerting. The ksqlDB UDFs/UDTFs (`containsPII`, `redactPII`, `extractPiiEntities`, `extractPiiEntityTypes`) provide flexibility for targeting specific fields and complex pipelining. The PII SMT for Kafka Connect removes sensitive information before data reaches Kafka brokers. These solutions support 25 entity types and can be deployed on the edge, on-premises, or in the cloud, integrating with Confluent Cloud's Stream Catalog and offering configurable deny lists and regex rules for custom entity detection.
  87. Build Streaming Data Pipelines Visually with Stream DesignerIntroduces Stream Designer, a visual interface for building, testing, and deploying streaming data pipelines natively on Kafka. It translates pipeline logic into ksqlDB code, integrates with Confluent connectors and Kafka topics, and allows switching between a graphical canvas and a SQL editor. It also supports importing existing ksqlDB pipelines and exporting them as SQL code.
  88. Getting Started with Stream Processing: The Ultimate GuideThis post introduces ksqlDB as a simplified stream processing solution, detailing its benefits for building real-time applications. It explains how ksqlDB simplifies architecture by handling event capture, transformations, aggregations, and materialized views within a single model, reducing infrastructure complexity. Technical use cases like streaming data pipelines, materialized caches, and event-driven microservices are explored. Key ksqlDB constructs such as Streams and Tables, Persistent Queries, and Push and Pull Queries are described, highlighting their role in processing data in motion and serving queries against it.
  89. Building a Real-Time Betting Platform with Confluent Cloud and AblyThis post demonstrates building a real-time betting platform by combining Confluent Cloud with Ably's edge messaging platform. It details the architecture where Confluent Cloud handles backend event streaming and processing (Kafka, ksqlDB), and Ably distributes real-time odds and receives bets from end-user applications (Android app) via WebSockets. The Ably Kafka Connector facilitates data transfer between Confluent Cloud and Ably. Key benefits highlighted include low latency, scalability, high availability, messaging QoS, ease of use, and increased engagement/revenue. The post outlines the setup of Confluent Cloud components (cluster, topics, ksqlDB) and the interaction with Ably for a continuous loop of real-time data processing and delivery.
  90. Real-Time Data Streams in the Cloud, On-Prem, or Both with ConfluentThis post introduces a tutorial demonstrating an end-to-end data pipeline using Confluent Cloud and Confluent Platform. It covers setting up Confluent Cloud and MongoDB Atlas, using the Datagen connector to stream sample data into Kafka, joining data sets into a combined stream using ksqlDB, and streaming the results into MongoDB Atlas. It also shows the same pipeline in a self-managed deployment using Confluent Platform and MongoDB Server, and uses Confluent Replicator to bridge the two environments.
  91. Real-Time Analytics with Apache Kafka and PinotThis post details the integration of Apache Pinot with Apache Kafka for real-time analytics. It explains how Pinot ingests data from Kafka topics, including schema and table configuration, real-time ingestion with mutable and immutable segments, and replication strategies. It also covers querying mechanisms via the Pinot broker, segment pruning through pre-partitioning (e.g., with ksqlDB), and memory management considerations for real-time Pinot tables to achieve low-latency, high-throughput analytical queries.
  92. Walmart’s Real-Time Inventory Management Powered by KafkaWalmart details their implementation of a real-time inventory management system using Apache Kafka. They address challenges with diverse data sources by implementing a canonical approach with a transformation engine to normalize data into a common inventory format. The post discusses scaling strategies, including adjusting Kafka partitions, tuning producer configurations (linger.ms, batch.size, acks=1) for latency and reliability, and scaling consumers based on partition count and server core availability. It also covers database optimization, specifically aligning Kafka partitioning with Cassandra partitioning to ensure single-consumer handling of item-store combinations and minimize deadlocks.
  93. Using Stream Processing to Prevent Fraud and Fight Account TakeoversThis post details building a fraud prevention system using Kafka Streams to analyze website login attempts in real-time, detecting new devices, new locations, botnet agents, and brute-force attacks. It leverages Kafka Streams' DSL for stream processing, including KStream and KTable operations, joins, filtering, and windowing. The post also discusses integration with other systems using Kafka Connect and provides a deep dive into the implementation of device recognition, including topology design and event mapping.
  94. Building an Event-Driven Stock Platform at EuronextEuronext has built a brand new market infrastructure and event-driven trading platform called Optiq with Confluent Platform at the core. They replaced their market data gateway with one that handles billions of messages per day, sending market data to vendors and trading members. Confluent Platform also enables them to build applications that interface with clearinghouses, monitor market latency, perform replication for disaster recovery, and store records in a data warehouse in compliance with regulatory requirements. The platform provides a tenfold increase in capacity to ingest messages and an average performance latency as low as 15 microseconds.
  95. Consuming Messages Out of Apache Kafka in a Browser | ConfluentThis post details the technical challenges and solutions for visualizing high-throughput Kafka message streams in a web browser. It explores pagination and long polling as initial approaches, highlighting their limitations. The primary solution presented is the use of WebSockets for real-time data streaming with minimal overhead. Performance issues encountered with Redux state management are discussed, along with optimizations achieved by switching to React component state and implementing batching. Finally, the use of Web Workers is introduced to offload WebSocket message parsing and rendering from the main UI thread, significantly improving browser responsiveness and throughput.
  96. We ❤️ syslogs: Real-time syslog processing with Apache Kafka and KSQL – Part 3: Enriching events with external dataThis post details enriching syslog data with external reference data from MongoDB using KSQL and Debezium. It covers streaming Ubiquiti device and user metadata from MongoDB into Kafka topics using Debezium, and then joining this reference data with the syslog stream in KSQL to enrich events with device names and IP addresses. The enriched data is then used for real-time analytics in Elasticsearch/Kibana and for triggering push notifications.