Confluent Connectors Portfolio

1/1/2026

This post introduces and elaborates on Confluent's extensive portfolio of over 120 pre-built connectors. It details the benefits of using these connectors over custom development or community options, focusing on modernization of technology stacks, increased developer productivity, and accelerated, de-risked time-to-value. It also provides a step-by-step guide for Confluent Cloud users to set up connectors and offers incentives for new users.

10/29/2025

Introduces Confluent Tableflow, a new capability that bridges Kafka streaming data to Databricks Unity Catalog by transforming Kafka topics into Delta Lake tables. It details the under-the-hood implementation using Kora and the Delta Kernel, including metadata fetching, reading from tiered storage, conversion to Parquet, committing metadata to Delta/Iceberg catalogs, and table maintenance. Highlights new features like schema evolution for Delta Lake, upserts for Delta tables, multi-format control, and multiple catalog integrations.

10/29/2025

Introduces General Availability for Delta Lake and Databricks Unity Catalog integration in Confluent Tableflow, enabling automated materialization of Kafka topics into Delta Lake tables with BYOS. Adds upsert materialization for CDC, allowing for "merge on read" tables using user-defined primary keys and handling updates/deletes. Enhances error handling with three strategies: Suspend, Skip, and Log to a Dead Letter Queue (DLQ) with standardized Avro schema for reprocessing and analysis. Implements Bring Your Own Key (BYOK) encryption for Delta Lake and Iceberg tables on AWS Dedicated clusters, supporting both Confluent-managed and customer-managed object storage with key permission checks for operational reliability. Expands Tableflow availability to Azure, allowing materialization into ADLS Gen2 and integration with Unity Catalog, Snowflake Open Catalog, and Apache Polaris.

10/22/2025

Introduced the Connect migration utility, an open-source CLI tool that automates the migration of self-managed Kafka connectors to Confluent's fully managed connectors. The utility provides phased migration capabilities including discovery of connector metadata, feasibility analysis for migration readiness, automated configuration mapping to fully managed equivalents, streamlined setup and creation of new connectors in Confluent Cloud, validation features for performance and data correctness comparison, and automated cutover and deployment options. This tool aims to transform connector migrations from a complex, manual process into a fast, transparent, and self-service experience.

10/8/2025

This post details how Confluent Tableflow addresses data lake governance challenges by shifting left, reinforcing data trust through schematization with Schema Registry, and providing efficient catalog management. It highlights the importance of metadata, schema evolution, and direct catalog synchronization to multiple catalogs (e.g., AWS Glue, Polaris, Databricks Unity) to minimize the mean time to analytics and ensure consistency across different compute engines and open table formats.

10/6/2025

This post announces new and enhanced integrations for Confluent's data streaming platform, focusing on expanding the AI data landscape and enabling real-time data access across various systems. New partner integrations include Amazon SageMaker, AWS Glue, ClickHouse, Couchbase, Datadog, DiffusionData, Firebolt, Infoview, LittleHorse, MariaDB, Onibex, SingleStore, Weaviate, and YugabyteDB. Enhancements were made to the Amazon EventBridge sink connector and AWS Lambda's native support for Confluent Schema Registry. Tableflow now supports Oracle Autonomous Database (ADB) and has improved metadata publishing service leveraging Schema Registry for Iceberg and Delta Lake. WarpStream has a new Datadog integration for monitoring.

9/30/2025

This post details best practices for data transformation, schema management, data masking/encryption, and data enrichment using Kafka Connect. It emphasizes leveraging Exactly-Once Semantics (EOS), transactional APIs, idempotent operations, two-phase commit strategies, and Schema Registry for robust data integration. Specific technical details include configurations for InsertField SMT, recommendations for Avro/Protobuf/JSON Schema formats, and compatibility levels (BACKWARD, FORWARD, FULL).

9/16/2025

This post details the integration of Confluent Cloud with AWS Marketplace, emphasizing streamlined procurement and the ability to leverage existing AWS commitments. It highlights key features of Confluent Cloud, including serverless scaling, integrated Apache Flink for stream processing, Apache Iceberg integration via Tableflow, over 120 pre-built connectors for AWS services, Schema Registry and Stream Governance, pay-as-you-go pricing with a 99.99% uptime SLA, and flexible deployment options. The post also touches on how developers, platform teams, and architects benefit from these capabilities, leading to faster real-time application development and reduced operational overhead.

8/20/2025

Introduces PII protection strategies within Kafka using Schema Registry data contracts. Demonstrates ensuring data quality of PII fields with CEL expressions, capturing invalid data in Dead Letter Queues (DLQs), performing simple masking of PII using CEL functions, and encrypting PII with client-side field-level encryption (CSFLE).

8/15/2025

This post details several enhancements to Confluent's fully managed connectors, focusing on improving the migration experience from self-managed to fully managed connectors and increasing flexibility. Key contributions include: exposing fully managed connector logs to non-admin roles (ResourceOwner, ConnectManager) for independent management and troubleshooting; introducing plugin versioning for custom connectors via API, CLI, and Terraform to enable seamless upgrades without losing offsets; expanding Single Message Transform (SMT) support with new out-of-the-box SMTs (DropHeaders, InsertHeader, HeaderFrom, ValueToKey, HeaderToValue, Flatten, Filter) and enabling custom SMTs with fully-managed connectors; and allowing custom naming for Dead Letter Queue (DLQ), success, and error topics, aligning fully managed connector functionality with self-managed offerings. Additionally, it lists specific enhancements across various connector categories, including database, SaaS, messaging, file storage, observability, and analytics connectors.

8/14/2025

This post introduces the Confluent OEM Program, which allows Independent Software Vendors (ISVs) to embed Confluent's enterprise-grade data streaming platform directly into their products. It highlights how this accelerates time to market, simplifies operations, and reduces risk for ISVs by leveraging Confluent Platform's capabilities, including Kafka, Flink, Control Center, KRaft, and Client-Side Field Level Encryption. The post also mentions WarpStream as an alternative for cloud object store integration and details the benefits of the OEM program such as expert support, validation, flexible commercial terms, and warranty.

7/29/2025

This post introduces Confluent Tableflow as a mechanism to materialize Kafka topics into open table formats like Iceberg and Delta Lake, effectively bridging operational and analytical estates. It contrasts Tableflow with traditional ETL and Zero-ETL, emphasizing Tableflow's benefits in data reusability, loose coupling through data contracts, and simplified connection management compared to MxN ETL connections. The post details how Tableflow leverages Kafka topics as a source for materialized tables, allowing for both streaming and table-based access to data, and discusses its advantages over Zero-ETL in scenarios requiring schema evolution and transformation before materialization.

7/11/2025

This post details the integration of Confluent Tableflow with the AWS Glue Data Catalog, enabling real-time analytics by materializing Kafka topics into Iceberg tables stored in Amazon S3. It explains how Tableflow handles data conversion, schema evolution, and table maintenance, and how the AWS Glue Data Catalog facilitates discovery and integration with AWS analytics services like Athena and Redshift.

7/1/2025

Introduces the Amazon EventBridge Sink Connector, enabling Confluent Platform to stream data to AWS EventBridge. Details setup instructions for AWS resources and Confluent Platform, including connector installation and configuration. Provides best practices for production, such as aligning service quotas, implementing dead-letter queues, enforcing least-privilege IAM, handling large event payloads via S3 offloading, and leveraging logging for debugging.

6/13/2025

This post details the integration of Confluent's data streaming platform with Databricks to build an AI-powered marketing personalization engine. It highlights the use of the Oracle XStream CDC Source Connector for ingesting data, Apache Flink for stream processing and AI model inference, and Confluent Tableflow for seamless synchronization of Kafka data as Delta Lake tables in Databricks. The post also introduces a quickstart guide for building this specific AI application, demonstrating how Confluent facilitates real-time data flow to analytical and AI platforms.

6/11/2025

This post announces the integration of Confluent Tableflow with Delta Lake (Open Preview) and Databricks Unity Catalog (Open Preview). It details how Tableflow materializes Kafka topics into Delta Lake tables, handling data conversion, schema evolution, catalog syncing (including Databricks Unity Catalog), and table maintenance. The integration with Unity Catalog provides centralized data discovery, access control, end-to-end lineage, and unified governance for streaming data within the Databricks Lakehouse Platform. The architecture involves configuring Tableflow to write to an S3 bucket as Delta Lake tables and then integrating with Unity Catalog by registering the S3 location as an External Location in Databricks and providing Databricks workspace details and service principal credentials to Tableflow.

6/4/2025

This post announces the General Availability (GA) of Confluent Tableflow for Apache Iceberg, detailing its integration with Snowflake Open Catalog. It explains how Tableflow materializes Kafka topics into Iceberg tables stored in object storage (e.g., S3) by converting Kafka segments and schemas (Avro, JSON, Protobuf) into Iceberg-compatible schemas and Parquet files, leveraging Confluent Schema Registry. Key capabilities highlighted include automated schema evolution, catalog syncing with Snowflake Open Catalog, table maintenance (compaction, snapshot expiration), and flexible storage options. The post also describes the process of configuring Tableflow and integrating it with Snowflake Open Catalog for seamless data discovery and querying by analytical engines like Snowflake Cortex AI.

5/20/2025

Introduces Snapshot Queries in Confluent Cloud for Apache Flink®, enabling batch-style queries across Kafka topics, Apache Iceberg, and Delta Lake tables via Tableflow. This allows for unified stream and batch processing using the same Flink SQL interface, with optimizations for interactive speeds on Tableflow. Also adds User-Defined Functions (UDFs) in Java for Flink on Azure, and enhances the Confluent for VS Code extension to support Flink development. Tableflow now supports Delta Lake tables in Open Preview, with planned integration for Databricks Unity Catalog. Introduces Confluent Cloud Network (CCN) routing for Flink and Schema Registry, IP Filtering for Flink and Schema Registry, and AWS PrivateLink for Schema Registry. Enterprise clusters are now available on Google Cloud, and Cross-Cloud Cluster Linking is in Limited Availability for replicating data between Kafka clusters on AWS and Azure.

4/22/2025

This post introduces strategies for achieving data efficiency by "shifting left" data governance and processing. It highlights the importance of schemas for data consistency and quality, and the potential of stream processing (using Apache Flink) for efficient data transformation. It also emphasizes the role of open table formats like Apache Iceberg for data reusability and consistency. Three key strategies are presented: implementing no-copy and zero ETL solutions, shifting data governance left, and reducing data waste by processing data at the source.

4/16/2025

This post introduces strategies for secure inter-organizational data sharing using Confluent's platform, detailing methods like Stream Sharing and Cluster Linking to address security and reliability challenges. It highlights how Confluent enables secure, scalable solutions for real-time data exchange across organizational boundaries, focusing on data producers' security concerns and data recipients' reliability requirements.

4/2/2025

Introduces the concept of data products and data contracts as a mechanism to shift data quality and governance left, enabling application engineers to take more responsibility for data quality at the point of creation. Details how data contracts, human- and machine-readable documents capturing metadata like schema, SLAs, and ownership, are published to Confluent Schema Registry. Explains how these contracts drive Kafka topics as interfaces and how Tableflow can materialize data into tabular formats, maintaining schema and metadata for analytical use cases.

3/19/2025

Tableflow is now generally available, representing Kafka topics and schemas as Apache Iceberg or Delta Lake tables. Freight clusters are generally available on AWS, offering cost savings for high-throughput, relaxed-latency workloads by writing directly to object storage. Enterprise clusters on AWS now support Bring Your Own Key (BYOK) and have increased maximum capacity to 32 eCKUs. Apache Flink® AI enhancements include Flink Native Inference for open-source and fine-tuned models, Flink Search for querying databases within Flink SQL for RAG, and built-in ML functions for time-series forecasting and anomaly detection. Confluent for VS Code is now generally available.

3/19/2025

This post announces the General Availability of Confluent Tableflow for materializing Kafka topics as Apache Iceberg tables. It details the challenges of streaming Kafka data into data lakes (fragility, data duplication, schema headaches, slow and expensive processing) and how Tableflow solves these by converting Kafka segments and schemas into Iceberg-compatible schemas and Parquet files. It highlights behind-the-scenes operations including data conversion using Schema Registry, automatic schema evolution detection, catalog syncing with AWS Glue, Snowflake Open Catalog, and Apache Polaris, and automatic table maintenance like file compaction and snapshot expiration. The post also mentions Delta Lake tables are in Open Preview and discusses the synergy with Apache Flink for stream processing before data lands in tables. It emphasizes native integrations with AWS, Databricks, and Snowflake catalogs and analytical engines.

3/7/2025

Introduces new partner integrations with Amazon Redshift, SAS, and Vectara to power real-time AI and analytics with streaming data. Amazon Redshift integration enables secure querying of Confluent Cloud data with mTLS. SAS integration connects Confluent with SAS Event Stream Processing for high-performance analytics. Vectara integration accelerates real-time AI assistants with streaming data. Updates to HiveMQ and Onibex integrations are also mentioned, with Onibex supporting real-time SAP data transfer to Snowflake with idempotent writes and schema evolution.

2/11/2025

This post details the expanded partnership between Confluent and Databricks, focusing on the bidirectional data flow between Confluent's Tableflow and Databricks' Unity Catalog. It highlights how this integration enables real-time, governed data products to power AI applications by transforming Kafka event streams into Delta Lake tables for AI model training and serving, and subsequently feeding AI-driven insights back into real-time applications. The post emphasizes unified governance and access across real-time and AI-driven data through the integration of Databricks Unity Catalog and Confluent Stream Governance.

2/4/2025

Introduces and details the fully managed HTTP Source V2 and HTTP Sink V2 connectors, enabling integration with any system exposing an HTTP API. Highlights new features such as configuring multiple APIs with the same hostname/auth, OpenAPI specification support, API testing during configuration, offset management, and client-side field-level encryption. HTTP Sink V2 also supports specifying Kafka record fields in the request URL, parameters, and body. Provides a detailed example of using these connectors for a fraud detection use case involving Stripe, Apache Flink, Azure OpenAI, and Pinecone.

1/6/2025

This post details how Reworkd leverages Confluent's Data Streaming Platform for real-time web scraping and processing for GenAI applications. Key technical contributions include: streaming scraped data into Kafka topics for validation and transformation, using Schema Registry for data governance and consistency, implementing shift-left stream processing for deduplication and data validation, and integrating Kafka with ClickHouse via Kafka Connect and Debezium for change data capture (CDC) into a data warehouse. The post also describes how Reworkd uses AI agents to automate web scraping code generation and validation, feeding the processed data through Confluent for near-real-time availability via a REST API.

12/13/2024

This post highlights the breadth of Confluent's connector ecosystem, emphasizing its role in connecting data silos and enabling data product creation. It mentions over 120 pre-built and 80+ managed connectors, along with support for custom connectors, to facilitate data integration across various sources and sinks.

12/5/2024

Introduces integration of Confluent Tableflow with Amazon SageMaker Lakehouse, enabling materialization of Kafka topics into Apache Iceberg tables stored in S3. Leverages AWS Glue Data Catalog for Iceberg table metadata management, allowing consumption by AWS analytics engines like Amazon Athena, Amazon Redshift, and Amazon EMR, as well as other compatible tools like Apache Spark. Demonstrates a use case for capturing CDC streams from PostgreSQL into Kafka and materializing them as Iceberg tables for downstream analytics.

12/4/2024

This post details the integration of Confluent Cloud and Amazon Redshift for streaming data into Redshift materialized views using mutual TLS (mTLS) authentication. It covers the setup process for both Confluent Cloud dedicated clusters and Amazon Redshift, including configuring private certificate authorities, setting up AWS client certificates, creating IAM policies and roles, and finally, configuring Redshift to consume from Confluent Cloud via mTLS.

11/20/2024

This post details the technical steps and configuration required to integrate AWS IoT Core with Confluent Cloud. It covers setting up Confluent Cloud with API keys and topics, configuring AWS Secrets Manager to store Confluent credentials, establishing an AWS IoT Core destination for Kafka, and creating an AWS IoT Core rule to route MQTT messages to a Confluent Cloud Kafka topic. It also includes optional debugging steps using error actions and AWS IoT Core logging.

10/25/2024

This post introduces the concept of a headless data architecture, which formalizes a data access layer at the center of an organization using both streams (Kafka) and tables (Iceberg). It details the problems with traditional multi-hop and medallion architectures (slowness, cost, brittleness, data sprawl) and proposes a shift-left approach. The post explains how to build this architecture using data products composed of a Kafka stream and an Iceberg table, with a focus on materializing streams into tables using Kafka Connect or Confluent's Tableflow. It also outlines a modular, incremental process for selecting data sets and shifting them left, and discusses considerations for stream and table evolution management.

10/17/2024

Introduces the concept of a headless data architecture, explaining its principles and benefits. Details how Apache Kafka and Apache Iceberg form the core of this architecture, enabling data streams and tables to be accessed independently by various processing engines. Discusses the role of schema registries and metadata catalogs for streams, and Iceberg's components (storage, catalog, transactions, time travel) for tables. Highlights benefits like reduced data duplication, cost savings, and flexibility in choosing processing tools. Contrasts headless architecture with data lakes and provides guidance on implementation.

10/11/2024

This post details strategies for handling bad data in event streams, focusing on prevention through schemas and tests, and remediation through event design and data recovery. It introduces the concept of State vs. Delta events, highlighting how State events, when combined with Kafka's topic compaction, enable effective deletion and correction of bad data. The post provides examples of fixing bad data at the source using Kafka Connect with Debezium for database changes, and within Kafka Streams applications, emphasizing the use of Dead Letter Queues (DLQs) for unrecoverable events.

10/8/2024

Introduces AWS IAM role integration for Confluent Cloud managed connectors, allowing the use of temporary security credentials instead of long-term access keys. This enhances security by reducing the attack vector and operational overhead associated with key rotation. The post details the process of creating an IAM role, setting up the Confluent Cloud Provider Integration, updating the IAM role trust policy, and finally creating a connector that utilizes the assumed IAM role, using Amazon DynamoDB CDC as a concrete example.

9/27/2024

Introduces the ability to manage Confluent resources (Topics, Clusters, Environments) and Confluent connectors (Azure Blob Storage Sink, Azure Blob Storage Source) directly from the Azure portal. This integration aims to simplify setup and management for enterprise customers processing real-time data on Azure.

8/30/2024

This post details how Busie, a startup, built its Charter Platform using Confluent Cloud. It highlights their migration from a monolithic to a microservices architecture, emphasizing the adoption of an event-driven approach with Apache Kafka. Busie leveraged Confluent Cloud's managed connectors to ingest data from 13 PostgreSQL and MongoDB databases, breaking down data silos. They also utilized stream processing to create a unified data model for their 'Charter Experience Finder' (ChEF) search engine, significantly reducing load times and enabling features like dynamic pricing and real-time notifications. The post also mentions future plans to use Flink for real-time analytics and yield-based pricing.

8/27/2024

Introduces a fully managed Confluent Cloud source connector for Amazon DynamoDB Change Data Capture (CDC). This connector simplifies data integration by handling infrastructure and software management, allowing users to focus on use cases. Key features include multi-table support, cost-effectiveness through shared tasks, flexible operational modes (SNAPSHOT, CDC, SNAPSHOT_CDC), enhanced developer productivity via TAG_MODE and INCLUDELIST_MODE for table discovery, custom offset support for precise stream positioning (TRIM_HORIZON, LATEST, AT_TIMESTAMP), and IAM AssumeRole for secure, temporary access to AWS resources.

8/19/2024

This post details the growth and usage statistics of Confluent's connector portfolio, highlighting top connectors by throughput (Amazon S3, Snowflake, GCS, BigQuery), popular connector combinations (MongoDB Atlas, PostgreSQL CDC, Snowflake, S3, BigQuery), and commonly used Single Message Transforms (SMTs) like InsertField, TopicRegexRouter, ReplaceField, ExtractField, and TimestampConverter. It also identifies the top connectors for initial deployments (Datagen, MongoDB Atlas, MySQL CDC, Snowflake) and the most downloaded connectors on Confluent Hub (JDBC, Debezium CDC, Apache Iceberg Sink).

8/6/2024

This post details the integration of Confluent Cloud with BigQuery's new continuous queries feature. It explains how BigQuery continuous queries can export data to Pub/Sub, which is then ingested by Confluent's Google Cloud Pub/Sub Source Connector into Kafka. This enables real-time data streaming from BigQuery into Confluent Cloud for further processing and downstream applications, facilitating use cases like real-time analytics and anomaly detection. The post provides a step-by-step guide on setting up this integration, including BigQuery query configuration and Confluent Cloud connector setup.

6/25/2024

This post details how Confluent's data streaming platform, specifically its connectors and stream processing capabilities (Flink), can be used for real-time log aggregation and analysis for cybersecurity. It highlights the challenges of traditional batch processing for security logs and presents a solution architecture using Confluent Cloud. The post includes example Flink SQL queries for detecting suspicious login attempts and correlating login data with IDS events, demonstrating the practical application of stream processing for security analytics. It also emphasizes the benefits of using connectors to ingest diverse log data and sink processed data to systems like Elasticsearch and BigQuery.

6/4/2024

This post highlights how Capital One leverages Confluent Cloud and Apache Kafka for data streaming, enabling real-time personalization, fraud detection, and a cloud-first banking infrastructure. It emphasizes the importance of data governance and managing data at scale to unlock AI/ML use cases and integrate legacy systems with cloud-native solutions. The post details Capital One's journey of migrating to the cloud and adopting Kafka to handle massive transaction volumes for 100 million customers.

5/21/2024

Introduces a new fully managed OpenSearch Sink Connector for Confluent Cloud, simplifying the process of streaming data from Kafka topics to OpenSearch indexes. Details the connector's architecture, configuration options (1:1, M:1, 1:M, N:M topic-to-index mappings), scalability through connector tasks, and cost considerations. Addresses limitations and workarounds for private networking with OpenSearch (PrivateLink, Transit Gateway, VPC peering) and notes the current lack of support for batch inserts and OpenSearch Serverless authentication, with ongoing work to address these.

5/18/2024

This post details how Moniepoint transformed its financial services by migrating from a legacy database system with replication lags to Confluent Cloud. It highlights the elimination of replication lags (up to 8-12 hours) by capturing transaction data changes efficiently, enabling near real-time data in read replicas. The post also describes the real-time migration of transaction data to a reporting data store without additional infrastructure costs. Furthermore, it explains how logging transactions on Confluent's managed streaming layer allowed for decoupling services, enabling independent consumption of events by financial solutions and resolving scalability issues, moving from hundreds of thousands to millions of requests. The implementation also addressed issues with database locks, low throughput, and scalability in a specialized microservice for transactions and notifications, achieving zero downtime for processing millions of requests.

4/23/2024

This post introduces the use of MQTT Source connectors within the Confluent platform to integrate IoT data from smart city applications. It details how Confluent, in conjunction with MQTT, can stream, process, and govern data from numerous clients in real-time. A specific example of an adaptive navigation system for emergency vehicles is provided, illustrating how data from CCTV, crowd-sourced reports, weather, and vehicle position is ingested via MQTT Source connectors, processed by Apache Flink, and used to provide real-time navigation updates.

3/19/2024

Introduces Tableflow, a new capability that unifies streaming and analytical data by materializing Kafka topics into Apache Iceberg or Delta Lake tables. It leverages Kora Storage Layer innovations to write Kafka segments to parquet files and uses a metadata materializer that taps into Schema Registry for Iceberg metadata and Delta transaction logs, handling schema mapping, evolution, and type conversions. Tableflow also includes continuous compaction of small files and currently supports the Iceberg REST catalog, with plans for AWS Glue and Polaris Catalog integrations, and direct customer-owned object storage.

3/12/2024

This post details how Confluent's Data Streaming Platform, including connectors and stream processing with Flink, can be applied to build real-time data pipelines for smart warehouses. It highlights the integration of IoT sensor data, inventory data from SQL Server and PostgreSQL, and robot status data. The post showcases how cluster linking enables hybrid cloud deployments and how Flink SQL can be used to process this data into actionable insights like low battery alerts and inventory counts. It also mentions the use of Confluent Connectors to ingest data from various sources and output to systems like Google BigQuery and GCS.

1/22/2024

Introduces the BigQuery Sink V2 Connector, a significant upgrade to the existing BigQuery Sink Connector. Key technical enhancements include the adoption of Google's Storage Write API for improved efficiency and cost-effectiveness, OAuth 2.0 support for enhanced security, parallel record streaming through an internal thread pool for greater scalability, and support for multiple data formats (Avro, JSON Schema, Protobuf, JSON). It also adds time-based table partitioning and invalid record routing to a DLQ. Migration recommendations from the legacy connector are provided.

11/28/2023

This post introduces new connectors and architectural patterns for core banking modernization. It details the MQ on Z connector for mainframe unlock without incurring MSU costs, and the Oracle CDC connector for Oracle unlock without GoldenGate licensing costs. It also highlights the use of Apache Flink for real-time data enrichment and MongoDB as a destination for modernized data. Architectural details are provided for the MQ on Z connector's direct access to MQ queues and the Oracle CDC connector's use of the Logminer API. The post also discusses cost savings associated with these connectors and provides considerations for their deployment.

8/29/2023

This post announces Confluent's Google Cloud Ready - Cloud SQL Designation, highlighting the integration of Confluent Cloud with Google Cloud SQL. It specifically mentions the MySQL Sink, PostgreSQL Sink, and PostgreSQL CDC Source connectors meeting functional and interoperability requirements for Cloud SQL. The post emphasizes how Confluent simplifies streaming pipelines to Google Cloud SQL using connectors, CDC, and SQL-based processing, enabling real-time data access and modernization for businesses.

5/25/2023

This post introduces a new DynamoDB source connector for Confluent Cloud, enabling Change Data Capture (CDC) from DynamoDB tables. It details two primary methods for achieving this: using DynamoDB Streams with AWS Lambda and a Kafka Producer, and leveraging Kinesis Data Streams. The post outlines the pros and cons of each method, focusing on fully managed services, end-to-end exactly-once semantics (with Kafka transactions), item-level ordering guarantees, and cost efficiency for the Lambda approach. It also highlights challenges with DynamoDB Streams' 24-hour retention limit and Kinesis' at-least-once delivery, and how Confluent's infinite storage and broader connector ecosystem address these limitations. The post provides a Python code snippet for the Lambda-based approach and discusses considerations for optimizing producer initialization and handling ephemeral Lambda functions.

12/6/2022

This post details the benefits and usage of Confluent Cloud's fully managed connectors, highlighting how they simplify Apache Kafka integration by offloading the operational burden of managing Kafka Connect workers and connector plugins. It emphasizes the speed, simplicity, and reliability of these connectors, which are configured through the Confluent Cloud UI or CLI. The post also introduces Stream Designer, a visual pipeline builder that integrates with fully managed connectors, and provides a demo of building a pipeline from Salesforce to Google BigQuery using these tools. Key features like single message transforms (SMTs), exposed connector logs, and data preview are also discussed.

9/20/2022

This post details the advantages and disadvantages of using JDBC source connectors versus Change Data Capture (CDC) for moving data from databases to Apache Kafka. It highlights CDC's ability to capture all events, including inserts, updates, and deletes, with lower database load and richer payload information. The post also presents a real-world use case of a legacy bank synchronizing two databases (PostgreSQL and SQL Server) using Debezium CDC connectors and Kafka Connect, and demonstrates how to prevent infinite loops using Single Message Transformations (SMTs) for filtering.

9/8/2022

This post details the technical steps and architecture for migrating data from an on-premise relational database (e.g., MySQL) to Amazon Aurora using Confluent Cloud. It covers setting up a MySQL Sink Connector to populate Aurora and a MySQL Source Connector to capture changes from Aurora. The post illustrates the process with screenshots of connector configuration and diagrams of the evolving architecture, highlighting the use of ksqlDB for data merging and consumers acting on events in real-time. It also discusses the challenges of binlog locations for migration and the need to recreate database objects in Aurora.

8/10/2022

This post introduces serverless stream processing by integrating ksqlDB with Azure Functions. It details two integration patterns: using the Confluent Azure Function sink connector and the Azure Function Kafka extension. The post examines the trade-offs between these methods, focusing on processing order guarantees, throughput, and the interaction between stateless Azure Functions and stateful ksqlDB. It provides code examples for ksqlDB queries and connector configurations, and discusses the internal workings of the Azure Function sink connector, including batching and concurrency.

7/28/2022

This post discusses the evolution of data pipelines, drawing parallels to the evolution of application development with microservices and DevOps. It highlights the 'pipeline problem' of fragility and lack of trust in traditional data pipelines. The post introduces two key trends: 'Data as a Product,' emphasizing decentralized ownership and consumer-centric data sharing, and 'Declarative Transformation Models,' exemplified by dbt, which promotes modular, testable, and reusable data transformation logic. It also touches on the importance of developer-friendly tooling and the adoption of software development lifecycle practices in data engineering.

4/27/2022

This post details how Confluent's platform, particularly its CDC connectors and Cluster Linking, accelerates cloud database modernization and migration by enabling real-time data movement from on-premises and multi-cloud environments to any cloud database. It highlights the shift from batch processing to real-time streaming, the benefits of managed services, and the reduction in Total Cost of Ownership (TCO). Specific CDC connectors for Oracle, Microsoft SQL Server, MySQL, PostgreSQL, and Salesforce are mentioned as key enablers for capturing database changes in real-time. Cluster Linking is presented as a mechanism to bridge on-premises Kafka clusters with Confluent Cloud for data migration. The post also references solution briefs and marketplace integrations for specific cloud providers (AWS Aurora, Azure Cosmos DB, Google Cloud SQL).

3/23/2022

This post details a pattern for deploying self-managed Kafka Connectors on AWS EKS Fargate, enabling a serverless operational model. It covers setting up IAM roles, configuring an EKS cluster with Fargate profiles, deploying the Confluent Operator, creating Kubernetes secrets for Confluent Cloud credentials, and deploying an S3 source connector using Confluent for Kubernetes. The post emphasizes abstracting operational burden for self-managed connectors.

1/19/2022

This post announces the expansion of Confluent Cloud's fully managed connectors portfolio to over 50, including new connectors for ActiveMQ, Databricks Delta Lake, Google BigTable, IBM MQ, MQTT, PagerDuty, RabbitMQ, Redis, SFTP, Splunk, and Zendesk. It details the benefits of fully managed connectors, such as eliminating operational burdens, reducing TCO, and enabling real-time data integration. The post also discusses the engineering journey to offer these connectors as fully managed services, abstracting away complexities of Connect worker management, internal topics, worker configs, monitoring, and security, while allowing for elastic scaling of task counts.

12/29/2021

This post details the integration of tcVISION, a data replication software product, with Confluent Cloud to enable real-time synchronization of mainframe data sources (Db2 z/OS, VSAM, IMS/DB, CA IDMS, CA Datacom, Software AG ADABAS) into Kafka topics. It outlines the setup process for Confluent Cloud, including account creation, cluster configuration (Basic cluster on AWS), and API key generation. It also describes the configuration of tcVISION using its control board and wizards to define metadata, replication scripts, and mappings for mainframe data to Confluent Cloud Kafka topics, specifically using JSON format. The post highlights tcVISION's change data capture (CDC) capabilities and its ability to handle data transformations and type conversions.

11/23/2021

Introduces the preview release of the Databricks Delta Lake Sink Connector for Confluent Cloud, enabling continuous real-time data synchronization from sources like Oracle databases to Databricks Delta Lake. Details the connector's functionality, including exactly-once semantics (EOS) support, and provides a step-by-step guide for setting up a CDC pipeline from Oracle to Delta Lake. Highlights the benefits of this integration for data warehouse modernization and real-time analytics.

10/14/2021

The Confluent Oracle CDC Source Connector now supports Oracle Database 19c. This involved adapting to the deprecation of LogMiner's 'continuous_mine' option by implementing a looping query mechanism for online and archived redo logs. Handling DDL statements like ALTER TABLE was improved by allowing LogMiner to use dictionaries from archived redo logs when the Online Catalog is insufficient, with configurable modes ('auto', 'online', 'redo_log') to manage this behavior. The post also includes a demonstration of a CDC pipeline from Oracle to Azure Synapse Analytics.

10/12/2021

Introduced Single Message Transforms (SMTs) to Confluent Cloud connectors, allowing in-line message key/value/header modifications. Added connector log events for enhanced visibility into connector operations and failures. Implemented a data preview feature for source connectors to validate output structure before deployment.

5/6/2021

Introduces and details the general availability of fully managed MongoDB Atlas Source and Sink Connectors for Confluent Cloud. Explains the setup and configuration process, including network access requirements for MongoDB Atlas and Confluent Cloud. Highlights key features of the source connector such as output formats (AVRO, JSON_SR, PROTOBUF), automatic schema registration with Schema Registry, and options for publishing full documents or specific changes. Describes the sink connector's functionality for writing Kafka data to MongoDB Atlas. Provides practical use case examples for both source and sink connectors, including loading sample restaurant data and propagating user information.

3/18/2021

This post details the creation and integration of a self-managed Azure Cosmos DB Connector for Kafka Connect. It covers the architecture for populating Azure Cosmos DB with data from Kafka using the Datagen Source Connector and the new Azure Cosmos DB Sink Connector. The post provides step-by-step instructions for environment setup, connector configuration (including topic-container mapping), and data validation in Azure Cosmos DB. It highlights the collaboration between Confluent and Microsoft's Commercial Software Engineering group.

3/2/2021

This post introduces two new Kafka Connectors: NettySource, which allows receiving data from network devices using TCP and UDP protocols, and PollableAPIClient, which enables pulling data from remote APIs. It details their configurations, including advanced options like TLS/SSL support, multiple tasks, health checks, and high availability deployments for NettySource. It also covers customization options for NettySource based on Netty's event model and discusses the architecture of PollableAPIClient, which separates data ingestion from Kafka Connect API details. Additionally, it mentions a Transformations library for data manipulation.

2/24/2021

This post introduces the Kafka Connect ODP Source Connector, a turnkey solution for integrating SAP systems with Apache Kafka. It details the connector's architecture, which leverages SAP Operational Data Provisioning (ODP) for data extraction from SAP NetWeaver AS ABAP-based systems like SAP ERP, ECC, and S/4HANA. The post highlights the benefits of using Kafka Connect, including scalability, performance, and seamless integration with Confluent Platform features like monitoring and exactly-once delivery semantics. It also discusses the use of SAP Java Connector (SAP JCo) for RFC-based connections and the support for Change Data Capture (CDC) through the Operational Delta Queue (ODQ) for near real-time data replication.

2/16/2021

Introduces the Confluent Oracle CDC Source Connector, enabling reliable and cost-effective continuous real-time data synchronization from Oracle Database to Confluent. Details its use of Oracle LogMiner, compatibility with Oracle versions 11g, 12c, 18c, and 19c, and configuration options for starting with a snapshot or from a specific SCN/timestamp. Highlights support for LOB data types and flexible numeric mapping. Provides a demo scenario and configuration for setting up a CDC pipeline between Oracle Database and BigQuery using the connector and the BigQuery Sink Connector.

2/9/2021

This post introduces detailed techniques for building Kafka connectors with proper configuration handling. It explains the use of `ConfigDef` for defining parameters, `ConfigDef.Validator` for ensuring value correctness, and `ConfigDef.Recommender` for providing dynamic, context-aware valid value suggestions. It also details how to override the `validate()` method to handle complex inter-configuration dependencies, which is a crucial aspect of connector development.

2/4/2021

This post demonstrates how to configure Azure Databricks to interact with Confluent Cloud for real-time data pipelines. It covers setting up Confluent Cloud (Kafka cluster, Schema Registry, Datagen Source connector) and Azure Databricks (workspace, cluster, libraries). The core technical contribution is the Python code for Spark ReadStream to consume Avro data from Confluent Cloud, parse it using Schema Registry, and store it on Azure Data Lake Storage (ADLS) in Delta Lake.

1/27/2021

This post details how Podium migrated their conversation search functionality from a Postgres read replica to Elasticsearch using a Kafka-based event streaming pipeline. It covers the use of Debezium PostgreSQL CDC Connector to ingest data into Kafka, ksqlDB for transforming and calculating conversation steps, joining records from different topics (users, conversations, conversation_items), and using tombstones to remove closed conversations from Elasticsearch. It also discusses factors like restreaming for idempotency and versioning topics for data evolution.

12/2/2020

This post introduces the 'Complete' aspect of Project Metamorphosis, focusing on Confluent's comprehensive ecosystem for event streaming. It highlights Confluent Hub with over 120 pre-built connectors for integrating existing systems with Kafka, including managed connectors in Confluent Cloud. Specific new connectors are listed: PostgreSQL Source (JDBC), MySQL Source (JDBC), Snowflake Sink, Amazon Redshift Sink, Oracle Database Source (JDBC), Microsoft SQL Server Source (JDBC), Google Cloud Spanner Sink, Azure Data Lake Storage Gen2 Sink, and Elasticsearch Service Sink. It also emphasizes ksqlDB for stream processing and turning data into action, detailing its capabilities for filtering, transforming, aggregating, creating materialized views, and enriching data. The post also shows how ksqlDB can manage connectors via SQL syntax, simplifying CDC setup.

10/30/2020

This post details how Qlik Replicate can be used to capture changes from databases in real-time using log-based Change Data Capture (CDC) and deliver them to Kafka. It covers the configuration of Qlik Replicate for Kafka targets, including broker connection, security, message format (JSON, Avro), message publishing strategies (topic naming, partitioning by key), and Schema Registry integration. It also discusses customizing message payloads to include header information and before/after data for updates, and provides a real-world example of its use in a credit card payment processing application.

9/18/2020

This post details how to connect to MongoDB Atlas in a secure environment by setting up VPC peering between Confluent Cloud and an AWS VPC, and establishing a PrivateLink between the AWS VPC and MongoDB Atlas. It provides step-by-step instructions for configuring VPC peering, including creating a dedicated Confluent Cloud cluster with VPC peering, setting up an AWS VPC and subnet, and accepting the peering request. It also outlines the process for setting up a PrivateLink between MongoDB Atlas and the AWS VPC, and creating a fully managed MongoDB Atlas source connector. The post emphasizes the security benefits of this approach, avoiding the need to whitelist broad IP ranges in MongoDB Atlas.

9/16/2020

This post details how to stream data from Apache Kafka on Confluent Cloud to Azure Data Explorer using the Kafka Connect Kusto Sink Connector. It explains the role of Kafka Connect for integration, the features of the Kusto Sink Connector (error handling, Schema Registry support, at-least-once delivery), and provides a step-by-step guide and hands-on lab example involving Spark on Azure Databricks, Confluent Cloud, and Azure Data Explorer.

9/1/2020

This post details the use of the Confluent Cloud CLI and the ccloud-stack utility for programmatically setting up Confluent Cloud services, including Kafka, Schema Registry, and ksqlDB. It also introduces the Datagen Source Connector for generating test data in various formats (Avro, JSON_SR, JSON, Protobuf) to topics, simplifying client application testing and demo building.

7/21/2020

This post introduces the integration of the Privitar Data Privacy Platform with Confluent Platform through the Privitar Kafka Connector. It details how this integration enables real-time de-identification of sensitive data in Kafka streams using predefined Privacy Policies, ensuring compliance with data privacy regulations. The post explains the Privitar Policy Manager, the application of various de-identification rules (e.g., redaction, tokenization, perturbation) to specific fields within messages, and demonstrates a de-identification scenario with JSON data. It also covers the optional re-identification capability and the deployment configuration for the connector.

6/26/2020

This post announces the general availability of the fully managed Snowflake sink connector in Confluent Cloud. It details how to configure the connector, including setting up Snowflake accounts with key pairs, and demonstrates a food delivery scenario where user action logs from Postgres are streamed to Kafka and then sunk to Snowflake for analysis. The post highlights the reduction in operational burden and the ability to create similar pipelines without managing Kafka Connect clusters.

6/23/2020

This post details how Confluent Cloud and its connector ecosystem can be used to unify disparate data pipelines during business mergers, using the Kinesis source connector as an example to ingest data into a central Kafka cluster. It highlights the use of ksqlDB for real-time metric computation on merged streams and the incremental merging of event streaming infrastructures.

3/27/2020

This post details the process of developing and contributing a custom Kafka connector to Confluent Hub, using the OpenSky Network Source Connector as a practical example. It elaborates on key aspects of connector development such as task parallelism for scalability, connector naming conventions, schema modeling with considerations for data translation and naming, the mechanism for committing offsets to ensure fault tolerance, and the importance of unit testing. Additionally, it outlines the steps for distributing the connector via Confluent Hub, including licensing considerations and the process for vendor verification. Finally, it demonstrates a complete workflow of a connector in action, integrating with ksqlDB and Elasticsearch for data visualization.

3/18/2020

This post demonstrates building a multi-cloud ETL pipeline using Confluent Cloud, specifically focusing on a log ingestion use case. It details the extraction of data from Amazon Kinesis using a managed Kinesis Source Connector, transformation using ksqlDB (counting events and summing durations), and loading into cloud storage (GCS, Azure Blob Storage, S3) via managed sink connectors. The post highlights the use of the Confluent Cloud CLI for managing these components and emphasizes the flexibility of Confluent Cloud for multi-cloud environments.

3/4/2020

This post details the Elasticsearch sink connector for Kafka Connect, explaining its purpose, installation via Confluent Hub, and configuration for transferring JSON data from Kafka to Elasticsearch. It covers common issues like deserialization errors due to incorrect converter configuration and demonstrates how to resolve them by specifying the JSON converter and handling schema and key ignoring. The post also explores using the connector with messages that have keys and schemas, and introduces Single Message Transforms (SMTs) like RegexRouter and TimestampRouter for modifying index names.

2/19/2020

This post announces the milestone of over 100 supported Kafka connectors and the revamp of Confluent Hub. It emphasizes the value of supported connectors for enterprise production deployments, contrasting them with self-developed or community connectors. The post details the criteria for Confluent Verified Integrations and lists popular connectors. It also introduces Confluent Platform 5.4 features like RBAC, Structured Audit Logs, Multi-Region Clusters, and Schema Validation, which enhance the overall Kafka ecosystem.

1/8/2020

This post details the architecture and benefits of offloading data from on-premises transactional databases to cloud analytics platforms using Confluent Platform and Kafka. It specifically demonstrates a pipeline for ingesting data from SQL Server and MySQL into Snowflake and Amazon S3 with Athena. Key technical components highlighted include Kafka Connect with Debezium SQL Server CDC Connector and Snowflake Kafka Connector. It also introduces stream processing with ksqlDB for data denormalization before loading into Snowflake, and discusses the advantages of event streaming over traditional batch ETL.

7/17/2019

This post introduces the official MongoDB Connector for Apache Kafka, developed and supported by MongoDB engineers and verified by Confluent. It details how this connector can be used as both a source (streaming data from MongoDB to Kafka topics using change streams with filtering) and a sink (streaming data from Kafka topics to MongoDB). The post provides installation instructions, configuration examples for both source and sink scenarios using Kafka Connect, and demonstrates testing with `kafkacat`. It highlights the use of MongoDB change streams for real-time event generation and the propagation of data from Kafka to MongoDB collections.

2/28/2019

Introduces the Kafka Connect Neo4j Sink plugin, detailing its installation, configuration, and testing with Docker Compose. It provides an example of a Cypher statement for ingesting data from a Kafka topic into Neo4j and demonstrates the ingestion of a large volume of events. It also introduces the `streams.publish` and `streams.consume` Neo4j Streams procedures for bidirectional data flow between Neo4j and Kafka.

2/12/2019

This post provides a deep dive into the JDBC Source Connector for Kafka Connect, detailing its setup, configuration options for various ingest strategies (bulk, incremental, query-based), and troubleshooting common issues. It emphasizes the importance of correct JDBC driver installation and JDBC URL specification, and outlines how to manage multiple tables and set Kafka message keys.

1/3/2019

Introduces the Kafka Connect Datagen Connector for generating test data, allowing users to create realistic datasets for testing client applications, building demos, and troubleshooting. It covers using the Confluent CLI to launch Confluent Platform and generate data with the Datagen connector, including customizing schemas and formats (Avro, JSON, String). Additionally, it demonstrates using the ksql-datagen command-line tool for generating Avro records directly to Kafka topics.

6/22/2018

Introduces Confluent Hub as a centralized platform for discovering, sharing, and installing Kafka Connectors, Transformations, and Converters. Launches the `confluent-hub` CLI tool for simplified installation of these components into Confluent Platform deployments. Details the types of components available on Confluent Hub (Connectors, Transformations, Converters) and the process for obtaining and installing them.

5/31/2018

Introduces the Kafka-Connect-Xenon connector, a Confluent-certified Kafka Connect sink that streams data into the Xenon analytics engine. Details Xenon's architecture for high-performance I/O and compute, its use cases for large-scale analytics, and its capabilities including SQL functionality, indexing, and multi-tenancy. Presents performance metrics of the connector ingesting 470GB of JSON data at 3.5 million messages per second on a single server with NVMe SSD, highlighting network as a bottleneck.

8/23/2017

This post details the second part of a series on building a Kafka Connect data pipeline, specifically focusing on streaming data from Kafka to Elasticsearch. It provides configuration details for the Elasticsearch sink connector, demonstrates loading the connector using the Confluent CLI, and shows how to query the data in Elasticsearch. It highlights that Kafka Connect sets the document ID in Elasticsearch to topic/partition/offset for exactly-once delivery and that the schema of the source MySQL table is used for mapping in Elasticsearch. It also mentions the use of Kibana for visualizing the data.

8/11/2017

This post introduces the Kafka Connect framework and demonstrates building a simple, no-code data pipeline. It details the use of the JDBC Source connector to ingest data from a MySQL database into Kafka, leveraging timestamp columns for change detection. The pipeline then streams this data to both a flat file and Elasticsearch. It highlights the distributed mode of Kafka Connect for scalability and ease of migration, and provides practical steps for setting up the MySQL database, configuring the JDBC connector, and verifying data flow using the Avro console consumer. It also touches upon the use of Kafka Connect offsets for tracking processed data.

2/28/2017

Introduces the Confluent Certified Kinetica Connector, which includes KineticaSourceConnector for streaming data from Kinetica to Kafka and KineticaSinkConnector for writing data from Kafka to Kinetica. Details the data formats (Struct) and supported data types for the sink connector. Highlights the integration with Kafka Connect API for fault tolerance and deployment via Confluent Control Center or RESTful API.

10/12/2016

This post details the integration of Oracle GoldenGate with Kafka Connect to stream Change Data Capture (CDC) events from Oracle databases into Kafka. It explains the setup process, including configuring Oracle GoldenGate's Extract and Replicat, the OGG-Kafka Connect adapter, and Kafka Connect properties. The post highlights the benefits of this approach over JDBC connectors, such as lower latency and reduced source database load. It also demonstrates streaming this data into Elasticsearch for search and analytics using the Confluent Elasticsearch Connector.

9/8/2016

Introduces the DataStax Certified Connector for Kafka to Cassandra, developed by DataMountaineer. Details its features including field selection, topic to table routing, error policies (RetriableException), and SSL support. Explains the use of Kafka Connect Query Language (KCQL) for configuration and demonstrates a practical use case with Confluent Platform, Cassandra Sink, and Cassandra. Provides configuration examples and steps for setting up and using the connector.

6/1/2016

Introduces a new Kafka Connect sink connector for PostgreSQL, detailing its implementation for streaming Kafka messages into PostgreSQL-compatible databases. Explains the connector's architecture, use cases, performance metrics (e.g., 200,000+ messages/sec), JSON parsing mechanism with path extraction, database connection configuration, and three delivery modes (fastest, guaranteed, synchronized) with a focus on achieving exactly-once semantics by storing Kafka offsets within database transactions. Also covers buffer size configuration and a setup walkthrough.