TechPrufer
Get started
Confluent
Confluent logoConfluent · Feature trail

Confluent Connectors Portfolio

99 posts · Feb 2016 – Oct 2025

What this trail covers

Confluent Cloud continues to evolve its Kafka Connectors portfolio, focusing on simplifying the migration of self-managed connectors to fully managed Confluent Cloud connectors. This iteration introduces new integrations with Amazon SageMaker, AWS Glue, ClickHouse, Couchbase, Datadog, DiffusionData, Firebolt, Infoview, LittleHorse, MariaDB, Onibex, SingleStore, Weaviate, and YugabyteDB. It also enhances existing integrations with Amazon EventBridge and AWS Lambda, and introduces Tableflow for exposing Kafka topics as open table formats (Iceberg/Delta Lake) for analytics engines. WarpStream also gains a Datadog integration for observability.

All Confluent trails
  1. Confluent Tableflow + Databricks Unity Catalog | Real-Time Data to AI-Ready TablesIntroduces Confluent Tableflow, a new capability that bridges Kafka streaming data to Databricks Unity Catalog by transforming Kafka topics into Delta Lake tables. It details the under-the-hood implementation using Kora and the Delta Kernel, including metadata fetching, reading from tiered storage, conversion to Parquet, committing metadata to Delta/Iceberg catalogs, and table maintenance. Highlights new features like schema evolution for Delta Lake, upserts for Delta tables, multi-format control, and multiple catalog integrations.
  2. Introducing Confluent Private CloudIntroduces General Availability for Delta Lake and Databricks Unity Catalog integration in Confluent Tableflow, enabling automated materialization of Kafka topics into Delta Lake tables with BYOS. Adds upsert materialization for CDC, allowing for "merge on read" tables using user-defined primary keys and handling updates/deletes. Enhances error handling with three strategies: Suspend, Skip, and Log to a Dead Letter Queue (DLQ) with standardized Avro schema for reprocessing and analysis. Implements Bring Your Own Key (BYOK) encryption for Delta Lake and Iceberg tables on AWS Dedicated clusters, supporting both Confluent-managed and customer-managed object storage with key permission checks for operational reliability. Expands Tableflow availability to Azure, allowing materialization into ADLS Gen2 and integration with Unity Catalog, Snowflake Open Catalog, and Apache Polaris.
  3. Migrate Kafka Connectors to Fully Managed—FastIntroduces the open-source Connect migration utility, a CLI-based tool that automates the migration of self-managed Kafka connectors to Confluent Cloud's fully managed connectors. The utility provides phased migration capabilities including discovery of existing connectors, feasibility analysis for migration readiness, mapping of configurations (including SMTs and authentication), setup and creation of managed connectors, validation of performance parity, and automated cutover. This significantly reduces the time and effort required for connector migrations.
  4. Better-Governed Data Lake Architectures with Confluent TableflowThis post details how Confluent Tableflow addresses data lake governance challenges by shifting left, reinforcing data trust through schematization with Schema Registry, and providing efficient catalog management. It highlights the importance of metadata, schema evolution, and direct catalog synchronization to multiple catalogs (e.g., AWS Glue, Polaris, Databricks Unity) to minimize the mean time to analytics and ensure consistency across different compute engines and open table formats.
  5. Expanding the AI Data Landscape: Confluent’s Q3 IntegrationsThis post announces new and enhanced integrations within the Confluent Connectors portfolio. New integrations include Amazon SageMaker (real-time model training), AWS Glue (cataloging and unifying streams), ClickHouse (real-time analytics), Couchbase (bidirectional data flow), DiffusionData (edge delivery), Firebolt (sub-second dashboards), Infoview (IBM i modernization), LittleHorse (workflow orchestration), MariaDB (CDC source), Onibex for ClickHouse (JDBC sink), SingleStore (real-time analytics/AI), Weaviate (vector database ingestion), and YugabyteDB (CDC events). Existing integrations with Amazon EventBridge and AWS Lambda are enhanced. Tableflow is highlighted for exposing Kafka topics as Iceberg/Delta Lake tables, with specific integrations for Snowflake Open Catalog, Apache Polaris, AWS Glue, Unity Catalog, and Oracle Autonomous Database. WarpStream gains a Datadog integration for monitoring.
  7. Best Practices for Kafka Connect Data Transformation & Schema ManagementThis post details best practices for data transformation and schema management within Kafka Connect. It covers ensuring data consistency through Exactly-Once Semantics (EOS), leveraging external systems' transactional APIs, implementing idempotent operations, and robust error handling with retries and dead-letter queues. It also outlines strategies for masking or encrypting sensitive data using Single Message Transforms (SMTs) and SSL/TLS, and emphasizes the importance of using Schema Registry with appropriate schema formats (Avro, Protobuf, JSON Schema) and compatibility levels (BACKWARD, FORWARD, FULL) for managing and validating schemas. Finally, it provides guidance on data enrichment using SMTs and optimizing data format conversions.
  8. Kafka Connect in Production: Scaling & Security GuideThis post provides a comprehensive guide to deploying Kafka Connect in production, detailing best practices for distributed mode architecture, Kubernetes deployment, scaling (workers and tasks), and throughput optimization. It emphasizes efficient data formats (Avro, Protobuf), batching, and the performance impact of Single Message Transforms (SMTs). Crucially, it outlines robust error handling patterns including backpressure management, Dead Letter Queues (DLQs) with context headers, and retries with idempotency. Security best practices cover encrypted transport (TLS), authentication (SASL, mTLS), authorization (ACLs), and secure credential management using Secret Providers. The post highlights the benefits of managed connectors on Confluent Cloud for simplifying these production concerns.
  9. How to Protect PII in Kafka With Schema Registry and Data ContractsIntroduces PII protection strategies within Kafka using Schema Registry data contracts. Demonstrates ensuring data quality of PII fields with CEL expressions, capturing invalid data in Dead Letter Queues (DLQs), performing simple masking of PII using CEL functions, and encrypting PII with client-side field-level encryption (CSFLE).
  10. Fully Managed Connector Updates for Easier MigrationsThis post details platform-level improvements for fully managed connectors, including exposing connector logs to non-admin roles (ResourceOwner, ConnectManager), introducing plugin versioning for custom connectors via CCPM APIs for seamless upgrades and offset preservation, and expanding Single Message Transform (SMT) support with new out-of-the-box SMTs (e.g., DropHeaders, InsertHeader, HeaderFrom, HeaderToValue, Flatten, Filter) and the ability to upload custom SMTs. It also introduces custom topic naming for DLQ, success, and error topics. Additionally, it lists specific enhancements across various connector types, such as PostgreSQL dialect support for Google Cloud Spanner sink, field truncation management for MongoDB Atlas connectors, automated publication for PostgreSQL CDC Source V2, proxy-based connectivity for Snowflake sink, and large object support for Oracle XStream CDC Source.
  11. Zero-ETL vs Tableflow | Integrating Apache Kafka® and Zero-ETL via Confluent TableflowThis post introduces Confluent Tableflow as a mechanism to materialize Kafka topics into open table formats like Iceberg and Delta Lake, effectively bridging operational and analytical estates. It contrasts Tableflow with traditional ETL and Zero-ETL, emphasizing Tableflow's benefits in data reusability, loose coupling through data contracts, and simplified connection management compared to MxN ETL connections. The post details how Tableflow leverages Kafka topics as a source for materialized tables, allowing for both streaming and table-based access to data, and discusses its advantages over Zero-ETL in scenarios requiring schema evolution and transformation before materialization.
  12. Unlocking Real-Time Analytics With Tableflow & AWS Glue CatalogThis post details the integration of Confluent Tableflow with the AWS Glue Data Catalog, enabling real-time analytics by materializing Kafka topics into Iceberg tables stored in Amazon S3. It explains how Tableflow handles data conversion, schema evolution, and table maintenance, and how the AWS Glue Data Catalog facilitates discovery and integration with AWS analytics services like Athena and Redshift.
  13. Integrate Kafka with AWS EventBridge Using ConfluentIntroduces the Amazon EventBridge Sink Connector, enabling Confluent Platform to stream data to AWS EventBridge. Details setup instructions for AWS resources and Confluent Platform, including connector installation and configuration. Provides best practices for production, such as aligning service quotas, implementing dead-letter queues, enforcing least-privilege IAM, handling large event payloads via S3 offloading, and leveraging logging for debugging.
  14. Build an AI Personalization Engine with Confluent & DatabricksThis post details the integration of Confluent's data streaming platform with Databricks to build an AI-powered marketing personalization engine. It highlights the use of the Oracle XStream CDC Source Connector for ingesting data, Apache Flink for stream processing and AI model inference, and Confluent Tableflow for seamless synchronization of Kafka data as Delta Lake tables in Databricks. The post also introduces a quickstart guide for building this specific AI application, demonstrating how Confluent facilitates real-time data flow to analytical and AI platforms.
  15. Delta Lake & Unity Catalog Now Integrate With TableflowThis post announces the integration of Confluent Tableflow with Delta Lake (Open Preview) and Databricks Unity Catalog (Open Preview). It details how Tableflow materializes Kafka topics into Delta Lake tables, handling data conversion, schema evolution, catalog syncing (including Databricks Unity Catalog), and table maintenance. The integration with Unity Catalog provides centralized data discovery, access control, end-to-end lineage, and unified governance for streaming data within the Databricks Lakehouse Platform. The architecture involves configuring Tableflow to write to an S3 bucket as Delta Lake tables and then integrating with Unity Catalog by registering the S3 location as an External Location in Databricks and providing Databricks workspace details and service principal credentials to Tableflow.
  16. Inside Tableflow GA: Real-Time Kafka to IcebergThis post announces the General Availability (GA) of Confluent Tableflow for Apache Iceberg, detailing its integration with Snowflake Open Catalog. It explains how Tableflow materializes Kafka topics into Iceberg tables stored in object storage (e.g., S3) by converting Kafka segments and schemas (Avro, JSON, Protobuf) into Iceberg-compatible schemas and Parquet files, leveraging Confluent Schema Registry. Key capabilities highlighted include automated schema evolution, catalog syncing with Snowflake Open Catalog, table maintenance (compaction, snapshot expiration), and flexible storage options. The post also describes the process of configuring Tableflow and integrating it with Snowflake Open Catalog for seamless data discovery and querying by analytical engines like Snowflake Cortex AI.
  17. Confluent Cloud Q2 '25 unifies stream and batch processingIntroduces Snapshot Queries in Confluent Cloud for Apache Flink®, enabling batch-style queries across Kafka topics, Apache Iceberg, and Delta Lake tables via Tableflow. This allows for unified stream and batch processing using the same Flink SQL interface, with optimizations for interactive speeds on Tableflow. Also adds User-Defined Functions (UDFs) in Java for Flink on Azure, and enhances the Confluent for VS Code extension to support Flink development. Tableflow now supports Delta Lake tables in Open Preview, with planned integration for Databricks Unity Catalog. Introduces Confluent Cloud Network (CCN) routing for Flink and Schema Registry, IP Filtering for Flink and Schema Registry, and AWS PrivateLink for Schema Registry. Enterprise clusters are now available on Google Cloud, and Cross-Cloud Cluster Linking is in Limited Availability for replicating data between Kafka clusters on AWS and Azure.
  18. Achieve Data Efficiency By Shifting Governance and Processing LeftThis post introduces strategies for achieving data efficiency by "shifting left" data governance and processing. It highlights the importance of schemas for data consistency and quality, and the potential of stream processing (using Apache Flink) for efficient data transformation. It also emphasizes the role of open table formats like Apache Iceberg for data reusability and consistency. Three key strategies are presented: implementing no-copy and zero ETL solutions, shifting data governance left, and reducing data waste by processing data at the source.
  19. Secure Inter-Organizational Data SharingThis post introduces strategies for secure inter-organizational data sharing using Confluent's platform, detailing methods like Stream Sharing and Cluster Linking to address security and reliability challenges. It highlights how Confluent enables secure, scalable solutions for real-time data exchange across organizational boundaries, focusing on data producers' security concerns and data recipients' reliability requirements.
  20. Improving Organizational Efficiency and Trust with Data ProductsIntroduces the concept of data products and data contracts as a mechanism to shift data quality and governance left, enabling application engineers to take more responsibility for data quality at the point of creation. Details how data contracts, human- and machine-readable documents capturing metadata like schema, SLAs, and ownership, are published to Confluent Schema Registry. Explains how these contracts drive Kafka topics as interfaces and how Tableflow can materialize data into tabular formats, maintaining schema and metadata for analytical use cases.
  21. Confluent Cloud Q1 ’25 introduces Tableflow, Freight clusters, and Flink AI enhancementsTableflow is now generally available, representing Kafka topics and schemas as Apache Iceberg or Delta Lake tables. Freight clusters are generally available on AWS, offering cost savings for high-throughput, relaxed-latency workloads by writing directly to object storage. Enterprise clusters on AWS now support Bring Your Own Key (BYOK) and have increased maximum capacity to 32 eCKUs. Apache Flink® AI enhancements include Flink Native Inference for open-source and fine-tuned models, Flink Search for querying databases within Flink SQL for RAG, and built-in ML functions for time-series forecasting and anomaly detection. Confluent for VS Code is now generally available.
  22. New with Confluent Platform 7.9Introduces the Oracle XStream CDC Source Connector for high-throughput, low-latency streaming of change events from Oracle databases. Adds Client-Side Field Level Encryption (CSFLE) to Confluent Platform (Early Access) for encrypting individual fields within messages on the producer side. Enhances security for ksqlDB and Flink with OAuth support. Updates Confluent Control Center (C3) with new consumer group management features. Announces General Availability of the official Confluent Node.js client for Apache Kafka. Introduces KRaft controller membership changes for dynamic quorum management. Confluent for Kubernetes (CFK) updates include dynamic PVC storage class migration (EBS to EFS), schema CRD enhancements, and HTTP proxy settings. CP Ansible updates include OIDC/OAuth 2.0 support for ksqlDB and Control Center, and RBAC enablement.
  23. Tableflow is now generally availableThis post announces the General Availability of Confluent Tableflow for materializing Kafka topics as Apache Iceberg tables. It details the challenges of streaming Kafka data into data lakes (fragility, data duplication, schema headaches, slow and expensive processing) and how Tableflow solves these by converting Kafka segments and schemas into Iceberg-compatible schemas and Parquet files. It highlights behind-the-scenes operations including data conversion using Schema Registry, automatic schema evolution detection, catalog syncing with AWS Glue, Snowflake Open Catalog, and Apache Polaris, and automatic table maintenance like file compaction and snapshot expiration. The post also mentions Delta Lake tables are in Open Preview and discusses the synergy with Apache Flink for stream processing before data lands in tables. It emphasizes native integrations with AWS, Databricks, and Snowflake catalogs and analytical engines.
  24. How Confluent and Databricks Are Unlocking Real-Time AIThis post details the expanded partnership between Confluent and Databricks, focusing on the bidirectional data flow between Confluent's Tableflow and Databricks' Unity Catalog. It highlights how this integration enables real-time, governed data products to power AI applications by transforming Kafka event streams into Delta Lake tables for AI model training and serving, and subsequently feeding AI-driven insights back into real-time applications. The post emphasizes unified governance and access across real-time and AI-driven data through the integration of Databricks Unity Catalog and Confluent Stream Governance.
  25. Connect Your HTTP APIs to Confluent with Fully Managed ConnectorsIntroduces the HTTP Source V2 and HTTP Sink V2 connectors, enhancing Confluent Cloud's ability to integrate with HTTP APIs. New features include support for multiple APIs per hostname, OpenAPI 3.0 configuration, API testing during configuration, offset management, and client-side field-level encryption. The HTTP Sink V2 also allows specifying topic, key, and record fields in the request URL, parameters, and body. The post provides a detailed example of using these connectors for a fraud detection use case involving Stripe, Flink, Azure OpenAI, and Pinecone.
  26. Reworkd Scales Real-Time Web Scraping With Data Streaming and GenAIThis post details how Reworkd leverages Confluent's Data Streaming Platform for real-time web scraping and processing for GenAI applications. Key technical contributions include: streaming scraped data into Kafka topics for validation and transformation, using Schema Registry for data governance and consistency, implementing shift-left stream processing for deduplication and data validation, and integrating Kafka with ClickHouse via Kafka Connect and Debezium for change data capture (CDC) into a data warehouse. The post also describes how Reworkd uses AI agents to automate web scraping code generation and validation, feeding the processed data through Confluent for near-real-time availability via a REST API.
  27. Materialize Your Topics with Tableflow and Amazon SageMaker LakehouseIntroduces integration of Confluent Tableflow with Amazon SageMaker Lakehouse, enabling materialization of Kafka topics into Apache Iceberg tables stored in S3. Leverages AWS Glue Data Catalog for Iceberg table metadata management, allowing consumption by AWS analytics engines like Amazon Athena, Amazon Redshift, and Amazon EMR, as well as other compatible tools like Apache Spark. Demonstrates a use case for capturing CDC streams from PostgreSQL into Kafka and materializing them as Iceberg tables for downstream analytics.
  28. Securely Streaming Data from Confluent into Amazon Redshift with mTLSThis post details the integration of Confluent Cloud and Amazon Redshift for streaming data into Redshift materialized views using mutual TLS (mTLS) authentication. It covers the setup process for both Confluent Cloud dedicated clusters and Amazon Redshift, including configuring private certificate authorities, setting up AWS client certificates, creating IAM policies and roles, and finally, configuring Redshift to consume from Confluent Cloud via mTLS.
  29. Bridge AWS IoT Core and Confluent Cloud to Contextualize Your IoT DataThis post details the technical steps and configuration required to integrate AWS IoT Core with Confluent Cloud. It covers setting up Confluent Cloud with API keys and topics, configuring AWS Secrets Manager to store Confluent credentials, establishing an AWS IoT Core destination for Kafka, and creating an AWS IoT Core rule to route MQTT messages to a Confluent Cloud Kafka topic. It also includes optional debugging steps using error actions and AWS IoT Core logging.
  30. Headless Data Architecture, Part 2: How to Build ItThis post introduces the concept of a headless data architecture, which formalizes a data access layer at the center of an organization using both streams (Kafka) and tables (Iceberg). It details the problems with traditional multi-hop and medallion architectures (slowness, cost, brittleness, data sprawl) and proposes a shift-left approach. The post explains how to build this architecture using data products composed of a Kafka stream and an Iceberg table, with a focus on materializing streams into tables using Kafka Connect or Confluent's Tableflow. It also outlines a modular, incremental process for selecting data sets and shifting them left, and discusses considerations for stream and table evolution management.
  31. Headless Data Architecture, Explained for DevelopersIntroduces the concept of a headless data architecture, explaining its principles and benefits. Details how Apache Kafka and Apache Iceberg form the core of this architecture, enabling data streams and tables to be accessed independently by various processing engines. Discusses the role of schema registries and metadata catalogs for streams, and Iceberg's components (storage, catalog, transactions, time travel) for tables. Highlights benefits like reduced data duplication, cost savings, and flexibility in choosing processing tools. Contrasts headless architecture with data lakes and provides guidance on implementation.
  32. Preventing and Fixing Bad Data in Event Streams, Part 2This post details strategies for handling bad data in event streams, focusing on prevention through schemas and tests, and remediation through event design and data recovery. It introduces the concept of State vs. Delta events, highlighting how State events, when combined with Kafka's topic compaction, enable effective deletion and correction of bad data. The post provides examples of fixing bad data at the source using Kafka Connect with Debezium for database changes, and within Kafka Streams applications, emphasizing the use of Dead Letter Queues (DLQs) for unrecoverable events.
  33. Assuming IAM Roles with Fully Managed ConnectorsIntroduces AWS IAM role integration for Confluent Cloud managed connectors, allowing the use of temporary security credentials instead of long-term access keys. This enhances security by reducing the attack vector and operational overhead associated with key rotation. The post details the process of creating an IAM role, setting up the Confluent Cloud Provider Integration, updating the IAM role trust policy, and finally creating a connector that utilizes the assumed IAM role, using Amazon DynamoDB CDC as a concrete example.
  34. Busie: Building a Charter Platform From the Ground Up with Confluent CloudThis post details how Busie, a startup, built its Charter Platform using Confluent Cloud. It highlights their migration from a monolithic to a microservices architecture, emphasizing the adoption of an event-driven approach with Apache Kafka. Busie leveraged Confluent Cloud's managed connectors to ingest data from 13 PostgreSQL and MongoDB databases, breaking down data silos. They also utilized stream processing to create a unified data model for their 'Charter Experience Finder' (ChEF) search engine, significantly reducing load times and enabling features like dynamic pricing and real-time notifications. The post also mentions future plans to use Flink for real-time analytics and yield-based pricing.
  35. Send to Confluent with the Managed DynamoDB CDC Source ConnectorThis post introduces the Confluent Cloud source connector for Amazon DynamoDB Change Data Capture (CDC). It highlights the benefits of this fully managed connector, including no-code integration, cost-effectiveness with multi-table support, multi-purpose usage (SNAPSHOT, CDC, SNAPSHOT_CDC), enhanced developer productivity through TAG_MODE and INCLUDELIST_MODE, custom offset support for precise stream positioning, and IAM AssumeRole for secure access. An example use case of an online purchase transaction is detailed, illustrating how CDC events from DynamoDB can power downstream processing and real-time analytics. The post also provides a step-by-step guide on setting up the connector in Confluent Cloud, including enabling DynamoDB streams, tagging tables, and configuring the connector with TAG discovery mode.
  36. BigQuery Continuous Queries & Confluent Yield Real-Time AnalyticsThis post details the integration of Confluent Cloud with BigQuery's new continuous queries feature. It explains how BigQuery continuous queries can export data to Pub/Sub, which is then ingested by Confluent's Google Cloud Pub/Sub Source Connector into Kafka. This enables real-time data streaming from BigQuery into Confluent Cloud for further processing and downstream applications, facilitating use cases like real-time analytics and anomaly detection. The post provides a step-by-step guide on setting up this integration, including BigQuery query configuration and Confluent Cloud connector setup.
  37. Enhanced Cybersecurity with Real-Time Log Aggregation and Analysis | ConfluentThis post details how Confluent's data streaming platform, specifically its connectors and stream processing capabilities (Flink), can be used for real-time log aggregation and analysis for cybersecurity. It highlights the challenges of traditional batch processing for security logs and presents a solution architecture using Confluent Cloud. The post includes example Flink SQL queries for detecting suspicious login attempts and correlating login data with IDS events, demonstrating the practical application of stream processing for security analytics. It also emphasizes the benefits of using connectors to ingest diverse log data and sink processed data to systems like Elasticsearch and BigQuery.
  38. Sending Events to OpenSearch with the New Confluent Sink ConnectorIntroduces a new fully managed OpenSearch Sink Connector for Confluent Cloud, simplifying the process of streaming data from Kafka topics to OpenSearch indexes. Details the connector's architecture, configuration options (1:1, M:1, 1:M, N:M topic-to-index mappings), scalability through connector tasks, and cost considerations. Addresses limitations and workarounds for private networking with OpenSearch (PrivateLink, Transit Gateway, VPC peering) and notes the current lack of support for batch inserts and OpenSearch Serverless authentication, with ongoing work to address these.
  39. How Moniepoint Transformed Financial Services for 2 Million Businesses in Africa with ConfluentThis post details how Moniepoint transformed its financial services by migrating from a legacy database system with replication lags to Confluent Cloud. It highlights the elimination of replication lags (up to 8-12 hours) by capturing transaction data changes efficiently, enabling near real-time data in read replicas. The post also describes the real-time migration of transaction data to a reporting data store without additional infrastructure costs. Furthermore, it explains how logging transactions on Confluent's managed streaming layer allowed for decoupling services, enabling independent consumption of events by financial solutions and resolving scalability issues, moving from hundreds of thousands to millions of requests. The implementation also addressed issues with database locks, low throughput, and scalability in a specialized microservice for transactions and notifications, achieving zero downtime for processing millions of requests.
  40. The Role of Data Streaming in Smart Cities | Confluent for IoTThis post introduces the use of MQTT Source connectors within the Confluent platform to integrate IoT data from smart city applications. It details how Confluent, in conjunction with MQTT, can stream, process, and govern data from numerous clients in real-time. A specific example of an adaptive navigation system for emergency vehicles is provided, illustrating how data from CCTV, crowd-sourced reports, weather, and vehicle position is ingested via MQTT Source connectors, processed by Apache Flink, and used to provide real-time navigation updates.
  41. Introducing Tableflow: Unifying Streaming and AnalyticsIntroduces Tableflow, a new capability that unifies streaming and analytical data by materializing Kafka topics into Apache Iceberg or Delta Lake tables. It leverages Kora Storage Layer innovations to write Kafka segments to parquet files and uses a metadata materializer that taps into Schema Registry for Iceberg metadata and Delta transaction logs, handling schema mapping, evolution, and type conversions. Tableflow also includes continuous compaction of small files and currently supports the Iceberg REST catalog, with plans for AWS Glue and Polaris Catalog integrations, and direct customer-owned object storage.
  42. Real-Time Data Streaming for Smart WarehousesThis post details how Confluent's Data Streaming Platform, including connectors and stream processing with Flink, can be applied to build real-time data pipelines for smart warehouses. It highlights the integration of IoT sensor data, inventory data from SQL Server and PostgreSQL, and robot status data. The post showcases how cluster linking enables hybrid cloud deployments and how Flink SQL can be used to process this data into actionable insights like low battery alerts and inventory counts. It also mentions the use of Confluent Connectors to ingest data from various sources and output to systems like Google BigQuery and GCS.
  43. Streamline Data Ingestion with Google BigQuery Sink V2 Connector on ConfluentIntroduces the BigQuery Sink V2 Connector, a significant upgrade to the existing BigQuery Sink Connector. Key technical enhancements include the adoption of Google's Storage Write API for improved efficiency and cost-effectiveness, OAuth 2.0 support for enhanced security, parallel record streaming through an internal thread pool for greater scalability, and support for multiple data formats (Avro, JSON Schema, Protobuf, JSON). It also adds time-based table partitioning and invalid record routing to a DLQ. Migration recommendations from the legacy connector are provided.
  44. Core Banking ModernizationThis post introduces new connectors and architectural patterns for core banking modernization. It details the MQ on Z connector for mainframe unlock without incurring MSU costs, and the Oracle CDC connector for Oracle unlock without GoldenGate licensing costs. It also highlights the use of Apache Flink for real-time data enrichment and MongoDB as a destination for modernized data. Architectural details are provided for the MQ on Z connector's direct access to MQ queues and the Oracle CDC connector's use of the Logminer API. The post also discusses cost savings associated with these connectors and provides considerations for their deployment.
  45. Kafka Connect: How to Increase Throughput on Source ConnectorsThis post provides detailed guidance on increasing throughput for Kafka Connect source connectors, focusing on tuning connector and producer configurations. It explains which components are tunable (connector, producer) and which are not (converter, transformations). Specific tuning parameters for connectors (e.g., `batch.max.rows`, `poll.interval.ms`) and producers (e.g., `batch.size`, `linger.ms`, `buffer.memory`, `compression.type`) are discussed with examples. It also outlines key JMX metrics for monitoring connector, broker, and producer performance to validate tuning efforts. A walk-through example using the JDBC Source connector demonstrates the process of gathering baseline throughput, modifying configurations, and observing the impact on metrics.
  46. How to Get Data from DynamoDB the Easy WayThis post details a method for capturing changes from DynamoDB using DynamoDB Streams and AWS Lambda, and publishing those changes to Confluent. It outlines the challenges with DynamoDB Streams (24-hour retention) and Kinesis Data Streams (at-least-once delivery) and presents a solution using Lambda to achieve end-to-end exactly-once semantics with Kafka transactions. It also discusses optimizing producer initialization for ephemeral Lambda functions and the cost-efficiency of using DynamoDB Streams with Lambda.
  47. How Fully Managed Connectors Make Apache Kafka EasierThis post introduces and elaborates on Confluent Cloud's fully managed connectors, which abstract away the operational burden of managing Kafka Connect workers and connector plugins. It highlights the benefits of reduced infrastructure costs, engineering time, and operational risks. The post details the ease of configuration through the Confluent Cloud UI/CLI, the breadth of the connector portfolio (over 70 connectors), and productivity features like Single Message Transforms (SMTs), exposed connector logs, and data preview. It also showcases the integration with Stream Designer for visual pipeline development, demonstrating a use case with Salesforce and Google BigQuery connectors.
  48. Sync Databases and Remove Data Silos with CDC & Apache KafkaThis post details the advantages and disadvantages of using JDBC source connectors versus Change Data Capture (CDC) for moving data from databases to Apache Kafka. It highlights CDC's ability to capture all events, including inserts, updates, and deletes, with lower database load and richer payload information. The post also presents a real-world use case of a legacy bank synchronizing two databases (PostgreSQL and SQL Server) using Debezium CDC connectors and Kafka Connect, and demonstrates how to prevent infinite loops using Single Message Transformations (SMTs) for filtering.
  49. Database Streaming with Confluent and Amazon AuroraThis post details the technical steps and architecture for migrating data from an on-premise relational database (e.g., MySQL) to Amazon Aurora using Confluent Cloud. It covers setting up a MySQL Sink Connector to populate Aurora and a MySQL Source Connector to capture changes from Aurora. The post illustrates the process with screenshots of connector configuration and diagrams of the evolving architecture, highlighting the use of ksqlDB for data merging and consumers acting on events in real-time. It also discusses the challenges of binlog locations for migration and the need to recreate database objects in Aurora.
  50. Serverless Stream Processing with Apache Kafka, Azure Functions, and ksqlDB | Confluent Serverless Stream Processing with Apache Kafka and AzureThis post introduces serverless stream processing by integrating ksqlDB with Azure Functions. It details two integration patterns: using the Confluent Azure Function sink connector and the Azure Function Kafka extension. The post examines the trade-offs between these methods, focusing on processing order guarantees, throughput, and the interaction between stateless Azure Functions and stateful ksqlDB. It provides code examples for ksqlDB queries and connector configurations, and discusses the internal workings of the Azure Function sink connector, including batching and concurrency.
  51. Modern Data Flow: The Key to Modern Data PipelinesThis post discusses the evolution of data pipelines, drawing parallels to the evolution of application development with microservices and DevOps. It highlights the 'pipeline problem' of fragility and lack of trust in traditional data pipelines. The post introduces two key trends: 'Data as a Product,' emphasizing decentralized ownership and consumer-centric data sharing, and 'Declarative Transformation Models,' exemplified by dbt, which promotes modular, testable, and reusable data transformation logic. It also touches on the importance of developer-friendly tooling and the adoption of software development lifecycle practices in data engineering.
  52. Accelerate Cloud Database Modernizations and Migrations with Confluent | Confluent Migrate Data to Any Cloud Database with ConfluentThis post details how Confluent's platform, particularly its CDC connectors and Cluster Linking, accelerates cloud database modernization and migration by enabling real-time data movement from on-premises and multi-cloud environments to any cloud database. It highlights the shift from batch processing to real-time streaming, the benefits of managed services, and the reduction in Total Cost of Ownership (TCO). Specific CDC connectors for Oracle, Microsoft SQL Server, MySQL, PostgreSQL, and Salesforce are mentioned as key enablers for capturing database changes in real-time. Cluster Linking is presented as a mechanism to bridge on-premises Kafka clusters with Confluent Cloud for data migration. The post also references solution briefs and marketplace integrations for specific cloud providers (AWS Aurora, Azure Cosmos DB, Google Cloud SQL).
  53. Deploy Serverless Apache Kafka Connectors on AmazonThis post details a pattern for deploying self-managed Kafka Connectors on AWS EKS Fargate, enabling a serverless operational model. It outlines the steps for setting up an EKS cluster with Fargate profiles, creating IAM roles for S3 access, deploying the Confluent Operator, configuring Kubernetes secrets for Confluent Cloud credentials, and deploying the S3 source connector using Confluent for Kubernetes (CFK). The example demonstrates reading data from S3 and writing it to a Confluent Cloud topic.
  54. Announcing the Confluent Q1 ‘22 Launch | ConfluentThis post announces the availability of over 50 fully managed connectors on Confluent Cloud, representing a significant expansion of the connector portfolio. It highlights the benefits of using fully managed connectors, including reduced operational burden, lower TCO, and accelerated data integration. Specific new connectors are listed, and the post emphasizes how these connectors enable real-time data pipelines to data warehouses, databases, and data lakes. It also touches upon the journey to fully managed connectors, contrasting them with self-managed Kafka Connect deployments and highlighting Confluent's expertise in building and managing these integrations. The post also mentions integrated monitoring with Datadog & Prometheus, dedicated cluster expand & shrink capabilities, single CSU pricing for ksqlDB, global schema linking, and the unified Confluent v2 CLI as part of the Q1 '22 launch.
  55. Unlock DB2 and Mainframe Data with tcVISION and ConfluentThis post details the integration of tcVISION, a data replication software product, with Confluent Cloud to enable real-time synchronization of mainframe data sources (Db2 z/OS, VSAM, IMS/DB, CA IDMS, CA Datacom, Software AG ADABAS) into Kafka topics. It outlines the setup process for Confluent Cloud, including account creation, cluster configuration (Basic cluster on AWS), and API key generation. It also describes the configuration of tcVISION using its control board and wizards to define metadata, replication scripts, and mappings for mainframe data to Confluent Cloud Kafka topics, specifically using JSON format. The post highlights tcVISION's change data capture (CDC) capabilities and its ability to handle data transformations and type conversions.
  56. Easily Migrate Data to the Cloud with Confluent and Databricks ConnectorIntroduces the Databricks Delta Lake Sink Connector for Confluent Cloud, enabling real-time data offloading from various sources (e.g., Oracle, Salesforce) to Databricks Delta Lake. The connector supports exactly-once semantics (EOS) by staging data in S3 before committing to Delta Lake. It details the configuration steps, including setting up the target Delta Lake table with a 'partition' INT field for EOS, and configuring connector details like host name, HTTP path, token, and S3 staging bucket credentials. The post highlights the benefits of reducing operational burden, accelerating migrations, and enabling real-time analytics.
  57. Confluent’s Oracle CDC Connector Now Supports Oracle Database 19cThis post announces support for Oracle Database 19c in the Oracle CDC Source Connector (v1.3.0). It details the technical challenges and solutions for handling Oracle 19c's desupported 'continuous_mine' option in LogMiner, requiring the connector to intelligently switch between online redo logs and archived logs for CDC data. It also addresses DDL statement handling by leveraging archived redo logs for LogMiner dictionaries when the online catalog is insufficient, and introduces configurable 'oracle.dictionary.mode' ('auto', 'online', 'redo_log') to manage this behavior. The post also provides a practical example of building a CDC pipeline from Oracle to Azure Synapse Analytics using this connector.
  58. Introducing New Connector Features and Single Message Transforms in Confluent CloudThis post introduces Single Message Transforms (SMTs) to Confluent Cloud's fully managed connectors, allowing for lightweight message modification (e.g., renaming fields, masking values, routing records) directly within the connector pipeline before data reaches Kafka or the sink system. It also introduces connector log events, exposing detailed operational logs to users for self-service troubleshooting of connector failures. Finally, it adds connector data output previews for source connectors, enabling users to validate data structure and format before actual ingestion without launching the connector.
  59. Error Handling Patterns in KafkaThis post introduces four error handling patterns for Kafka applications: Stop on Error, Dead Letter Queue, Add a Retry Topic and Retry Application, and Maintain Order of Redirected Events. It details the implementation and trade-offs of each pattern, including scenarios where order preservation is critical and how to manage it using retry and redirect topics with tombstone events. The post also highlights the potential for out-of-order processing with retry topics and provides a recovery mechanism for the in-memory store used in the order-preserving pattern.
  60. Confluent Cloud Offers Fully Managed MongoDB Connector for KafkaIntroduces the general availability of fully managed MongoDB Atlas Source and Sink Connectors within Confluent Cloud. Details configuration steps for network access between Confluent Cloud and MongoDB Atlas, including the use of static egress IPs. Provides walkthroughs for using the Source Connector to ingest data from MongoDB Atlas into Kafka topics (e.g., restaurant data) and the Sink Connector to write data from Kafka topics into MongoDB Atlas (e.g., user data). Highlights connector features like output formats (AVRO, JSON_SR, PROTOBUF), schema registration, and options for publishing full documents or specific changes.
  61. Integrate Azure and Confluent to Stream Data Into Cosmos DBThis post details the integration of Azure Cosmos DB with Confluent Platform using a self-managed Azure Cosmos DB Connector. It provides a step-by-step guide for setting up the environment, configuring the Datagen Source Connector and the Azure Cosmos DB Sink Connector, and validating data ingestion into Azure Cosmos DB. The post highlights the importance of topic-container mapping for the connector's functionality and demonstrates a practical use case for streaming data from Kafka to a NoSQL database on Azure.
  62. To Push or Pull Data into Kafka Connect? 3 Ways to Send Data to KafkaThis post introduces two new Kafka Connectors: NettySource for receiving data over TCP/UDP and PollableAPIClient for pulling data from APIs. It details their configurations, including TLS/SSL support for NettySource, and strategies for high availability deployment using HAProxy and Keepalived. It also mentions the Transformations library for message manipulation.
  63. Integrate Apache Kafka and SAP with the Kafka Connect ODP Source ConnectorThis post introduces the Kafka Connect ODP Source Connector, a turnkey solution for integrating SAP systems with Apache Kafka. It leverages SAP Operational Data Provisioning (ODP) to extract data from SAP NetWeaver ABAP-based systems (like SAP ECC and S/4HANA) and write it to Kafka. The connector is compliant with the Kafka Connect API, benefiting from Confluent Platform's features like centralized logging, monitoring, configuration, failover recovery, and exactly-once delivery semantics. It uses SAP JCo for RFC-based connections for optimal performance and supports delta replication and change data capture (CDC) through the Operational Delta Queue (ODQ). The installation is simplified via Confluent Hub and integrates with Confluent Control Center for management.
  64. Connect Kafka to Oracle with Confluent's Premium Oracle CDC ConnectorIntroduces Confluent's Oracle CDC Source Connector, enabling reliable and cost-effective continuous real-time syncs by offloading data from Oracle Database to Confluent Cloud. Details the connector's capabilities, version compatibility (Oracle 11g, 12c, 18c, 19c), and provides a demo scenario for setting up a CDC pipeline between Oracle Database and BigQuery using the Oracle CDC Source Connector and the BigQuery Sink Connector. Covers Kafka Connect worker configuration, Oracle database prerequisites for CDC, and connector configuration parameters, including topic naming templates and LOB handling. Highlights benefits like cost savings and simplified data pipelines. Mentions Kafka Connect's topic creation functionality for new topics.
  65. How to Write a Kafka Connector with Proper Configuration HandlingThis post details how to correctly handle configuration for Kafka Connectors. It explains the use of `ConfigDef` to define configuration parameters, including types, importance, and default values. It introduces `ConfigDef.Validator` for validating individual configuration values and `ConfigDef.Recommender` for suggesting valid values, especially when they depend on other configurations. The post also describes how the `validate()` method in the `Connector` class is used by Kafka Connect to validate configurations before instance creation and how to override it for complex inter-config dependencies. It highlights the role of these mechanisms in GUI applications like Confluent Control Center.
  66. Build Streaming Data Pipelines with Confluent, Databricks, and AzureThis post demonstrates how to configure Azure Databricks to interact with Confluent Cloud for real-time data pipelines. It covers setting up Confluent Cloud (Kafka cluster, Schema Registry, Datagen Source connector) and Azure Databricks (workspace, cluster, libraries). The core technical contribution is the Python code for Spark ReadStream to consume Avro data from Confluent Cloud, parse it using Schema Registry, and store it on Azure Data Lake Storage (ADLS) in Delta Lake.
  67. How Apache Kafka Enables Podium to "Ship It and See What Happens"This post details how Podium migrated their conversation search functionality from a Postgres read replica to Elasticsearch using a Kafka-based event streaming pipeline. It covers the use of Debezium PostgreSQL CDC Connector to ingest data into Kafka, ksqlDB for transforming and calculating conversation steps, joining records from different topics (users, conversations, conversation_items), and using tombstones to remove closed conversations from Elasticsearch. It also discusses factors like restreaming for idempotency and versioning topics for data evolution.
  68. Kafka Tools that Boost Developer ProductivityThis post introduces and details the usage of three command-line tools that enhance developer productivity when working with Apache Kafka: kafkacat for producing and consuming messages (including Avro with Schema Registry), Confluent REST Proxy for HTTP-based Kafka interactions (producing Avro, consuming, and metadata retrieval), and jq for processing JSON output from these tools. It demonstrates practical examples of using these tools for common development tasks.
  69. Project Metamorphosis Month 8: Complete Apache Kafka in Confluent CloudThis post announces the final launch of Project Metamorphosis, focusing on the 'Complete' aspect of Confluent Cloud. It highlights the availability of over 120 pre-built connectors for Kafka through Confluent Hub, enabling real-time integration of existing databases and systems. It specifically lists newly available managed connectors in Confluent Cloud for various databases (PostgreSQL, MySQL, Oracle, SQL Server) and data warehouses (Snowflake, Redshift, Azure Data Lake Storage Gen2, Google Cloud Spanner), as well as Elasticsearch. It also mentions Cluster Linking for global architectures. The post also emphasizes ksqlDB for turning data into action, detailing its capabilities for stream processing, materialized views, and managing embedded connectors via SQL syntax. It positions Confluent Cloud ksqlDB as a fully managed, easier-to-use alternative to Kafka Streams.
  70. Streaming Data Pipelines and Analytics with Qlik and KafkaThis post introduces Qlik Replicate as a tool for real-time Change Data Capture (CDC) from databases to Kafka. It details how Qlik Replicate monitors database transaction logs to capture changes with low impact, avoiding queries or triggers. The post outlines configuration options for the Kafka target, including broker connection, security models, message formats (JSON, Avro), message publishing (topic naming, partitioning strategy by message key or schema/table name), and Schema Registry integration. It also discusses customizing message payloads to include header information (operation type, timestamp) and before/after data values, and the ability to omit specific columns. A real-world example of a credit card payment processing application architecture using Qlik Replicate and Kafka is provided.
  71. Connect Kafka to New Relic for Real-Time Monitoring and ObservabilityIntroduces the New Relic connector for Kafka Connect, enabling the ingestion of Kafka metrics and events into the New Relic Telemetry Data Platform. Details configuration for both event and metric telemetry data types, including sample JSON payloads and required fields. Explains error handling mechanisms provided by Kafka Connect, such as retries, dead letter queues, and payload splitting for oversized messages. Provides guidance on installing the connector and obtaining New Relic API keys.
  72. How to Connect to MongoDB Atlas in a Secure EnvironmentThis post details how to connect to MongoDB Atlas in a secure environment by setting up VPC peering between Confluent Cloud and an AWS VPC, and establishing a PrivateLink between the AWS VPC and MongoDB Atlas. It provides step-by-step instructions for configuring VPC peering, including creating a dedicated Confluent Cloud cluster with VPC peering, setting up an AWS VPC and subnet, and accepting the peering request. It also outlines the process for setting up a PrivateLink between MongoDB Atlas and the AWS VPC, and creating a fully managed MongoDB Atlas source connector. The post emphasizes the security benefits of this approach, avoiding the need to whitelist broad IP ranges in MongoDB Atlas.
  73. Stream Data from Apache Kafka to Azure Data ExplorerThis post demonstrates how to stream data from Apache Kafka on Confluent Cloud to Azure Data Explorer using the Kafka Connect Kusto Sink Connector. It provides a practical example using the Chicago crimes dataset, detailing the setup steps for both Confluent Cloud and Azure Data Explorer, including provisioning clusters, creating topics and tables, and configuring the connector. The post also outlines the process of building and deploying a custom Kafka Connect worker image on Azure Kubernetes Service (AKS) to host the Kusto sink connector, as it is not yet a managed connector offering.
  74. Easy Ways to Test Your Application with Serverless KafkaThis post introduces two primary methods for testing applications with Kafka: running a local Confluent Platform cluster and using serverless Kafka in Confluent Cloud. It details the use of the Confluent Cloud CLI and the `ccloud-stack` utility to programmatically provision a Confluent Cloud environment, including Kafka, Schema Registry, and ksqlDB. It also demonstrates how to use the managed Datagen Source Connector to generate mock data for testing client applications, including configuration and deployment steps.
  75. Data Streaming Privacy, Security, and Compliance in KafkaThis post introduces the integration of the Privitar Data Privacy Platform with Confluent Platform through the Privitar Kafka Connector. It details how this integration enables real-time de-identification of sensitive data in Kafka streams using predefined Privacy Policies, ensuring compliance with data privacy regulations. The post explains the Privitar Policy Manager, the application of various de-identification rules (e.g., redaction, tokenization, perturbation) to specific fields within messages, and demonstrates a de-identification scenario with JSON data. It also covers the optional re-identification capability and the deployment configuration for the connector.
  76. Spring Kafka Beyond the Basics - How to Handle Failed Kafka ConsumersThis post details how to handle 'poison pill' scenarios in Spring Kafka consumers, specifically focusing on deserialization failures. It explains the causes of poison pills (incompatible serializers/deserializers), provides an example project demonstrating how to trigger and handle them, and outlines configuration strategies using Spring Kafka's error handling mechanisms, including the use of ErrorHandlingDeserializer and Dead Letter Queue (DLQ) topics for problematic messages. It also touches upon the importance of schema enforcement with Avro and Schema Registry.
  77. Fully Managed Snowflake Kafka Sink Connector in Confluent CloudThis post announces the general availability of the fully managed Snowflake sink connector in Confluent Cloud. It highlights the reduction in operational burden by eliminating the need to manage a Kafka Connect cluster. The post provides a practical example of using the connector with a food delivery scenario, demonstrating how to configure it and ingest data from a Kafka topic (SUM_PER_SOURCE) into Snowflake. It also includes setup instructions involving key pair generation for Snowflake authentication and mentions the importance of regional alignment between Confluent Cloud and Snowflake accounts to avoid data movement charges.
  78. Merging Data StreamsThis post demonstrates how to integrate data streams from Amazon Kinesis into Confluent Cloud using the fully managed Kinesis source Kafka connector. It outlines the steps involved: creating a Kinesis data stream, provisioning a Confluent Cloud cluster, creating a Kafka topic, configuring the Kinesis source connector, and producing messages to Kinesis to observe the data flow. This showcases a practical application of connectors for unifying disparate data streams during business mergers.
  79. How to Contribute a Kafka Connector on Confluent HubThis post details the process of developing and distributing a custom Kafka Connector for the OpenSky Network API. It covers key considerations for connector development, including task parallelism for scalability, naming conventions, configuration options (specifically SMT support), schema modeling and naming strategies, and offset committing mechanisms. It also highlights the importance of unit testing and the process of distributing connectors via Confluent Hub, including licensing considerations. Finally, it provides a practical example of deploying and visualizing the connector's output using ksqlDB and Elasticsearch.
  80. Build a Cloud ETL Pipeline - Multi-Cloud Integration with Confluent CloudThis post demonstrates building a multi-cloud ETL pipeline using Confluent Cloud, specifically focusing on a log ingestion use case. It details the extraction of data from Amazon Kinesis using a managed Kinesis Source Connector, transformation using ksqlDB (counting events and summing durations), and loading into cloud storage (GCS, Azure Blob Storage, S3) via managed sink connectors. The post highlights the use of the Confluent Cloud CLI for managing these components and emphasizes the flexibility of Confluent Cloud for multi-cloud environments.
  81. Kafka Elasticsearch Connector Tutorial with ExamplesThis post details the setup and configuration of the Elasticsearch Sink Connector for Kafka Connect, including how to transfer JSON data from Kafka topics to Elasticsearch. It addresses common issues like serialization errors due to incorrect converter configurations (Avro vs. JSON) and demonstrates how to update connector configurations using the REST API. The post also explores the use of Single Message Transforms (SMTs) like RegexRouter and TimestampRouter for modifying index names and handling message keys and schemas.
  82. Celebrating Over 100 Apache Kafka ConnectorsThis post announces the milestone of reaching over 100 supported connectors for Apache Kafka, highlighting the expansion of Confluent's connector portfolio. It emphasizes the value of these connectors for developers by contrasting building custom connectors with leveraging vendor-supported options. The post also details the revamp of Confluent Hub as an online marketplace for discovering and filtering connectors based on type, support, licensing, and cloud availability. It mentions the inclusion of popular connectors like Salesforce, InfluxDB, Google BigQuery, Azure Blob Storage, AWS Lambda, and Syslog.
  83. Pipeline to the Cloud - On-Premises Data Streaming for Cloud AnalyticsThis post details the architecture and benefits of offloading data from on-premises transactional databases to cloud analytics platforms using Confluent Platform and Kafka. It specifically demonstrates a pipeline for ingesting data from SQL Server and MySQL into Snowflake and Amazon S3 with Athena. Key technical components highlighted include Kafka Connect with Debezium SQL Server CDC Connector and Snowflake Kafka Connector. It also introduces stream processing with ksqlDB for data denormalization before loading into Snowflake, and discusses the advantages of event streaming over traditional batch ETL.
  84. Single Message Transforms in Kafka ConnectThis post introduces and details Single Message Transforms (SMTs) within Kafka Connect, explaining their purpose, common use cases, configuration, and how to write custom SMTs. It clarifies the role of SMTs in manipulating individual records within data pipelines and contrasts their capabilities with more complex stream processing tasks.
  85. 4 Steps to Creating Apache Kafka Connectors with the Kafka Connect APIThis post introduces the Kafka Connect API as a method for developing custom connectors when existing ones are insufficient. It explains the core concepts of Kafka Connect, including connectors, tasks, and configuration management. The article provides a four-step guide to developing a custom source connector, focusing on defining configuration properties using `AbstractConfig`, passing these properties to tasks via `taskConfigs`, implementing the `SourceTask` for data streaming logic, and finally, the `start` and `stop` methods for lifecycle management. It also touches upon running connectors in standalone vs. distributed modes and the role of the Kafka Connect REST API.
  86. Getting Started with the MongoDB Connector for Apache Kafka and MongoDB | ConfluentThis post details the integration of the MongoDB Connector for Apache Kafka, a verified integration. It provides instructions for installing and configuring the connector, demonstrating its use as both a source (monitoring MongoDB change streams to Kafka) and a sink (streaming data from Kafka topics to MongoDB). It includes example configurations for Kafka Connect and usage of kafkacat for testing, highlighting the use of MongoDB change streams and the flexibility of Kafka Connect converters.
  87. Building a Scalable Search Architecture | ConfluentThis post details the technical implementation of building a scalable search architecture using Kafka Connect. It explains how to use the JDBC Source Connector to incrementally ingest data from relational databases (MySQL example provided) into Kafka topics, covering different operational modes (incrementing, timestamp, combination, custom query) and the importance of JDBC drivers and schema evolution with Schema Registry. It then describes the setup and configuration of the Elasticsearch Sink Connector to index data from Kafka topics into Elasticsearch, highlighting considerations for mappings and dynamic fields. The post advocates for a Kafka-centric approach to decouple data producers and consumers, improving scalability and flexibility.
  88. Kafka Connect Deep Dive – Error Handling and Dead Letter Queues | ConfluentThis post introduces and details Kafka Connect's error handling mechanisms, specifically focusing on 'fail fast' behavior, silently ignoring errors, and routing messages to a Dead Letter Queue (DLQ). It provides concrete examples of how to configure these options using `errors.tolerance`, `errors.deadletterqueue.topic.name`, and `errors.deadletterqueue.context.headers.enable`. The post also illustrates how to inspect DLQ messages and highlights the importance of monitoring JMX metrics for error rates. It demonstrates the default behavior of Kafka Connect, the impact of setting `errors.tolerance = all`, and the benefits of using a DLQ for robust pipeline management.
  89. All About the Kafka Connect Neo4j Sink Plugin | ConfluentThis post announces the availability of the Kafka Connect Neo4j Sink plugin on Confluent Hub. It provides an overview of Neo4j as a graph database, details the functionality of the sink plugin for streaming Kafka events into Neo4j, and includes instructions for installation and configuration. It also demonstrates local testing with Docker Compose, including sample configurations and Cypher statements for data ingestion. Additionally, it introduces two Neo4j Streams procedures, `streams.publish` and `streams.consume`, for direct message streaming from Neo4j to Kafka and vice-versa.
  90. Kafka Connect Deep Dive – JDBC Source Connector | ConfluentThis post provides a deep dive into setting up and troubleshooting the JDBC Source Connector for Kafka Connect, focusing on self-managed scenarios. It covers essential configuration aspects such as JDBC driver installation and availability, specifying tables for ingestion, incremental ingest strategies (including query-based ingest), managing multiple connectors, and common troubleshooting scenarios like 'No suitable driver found' and 'Why is there no data?'. It also details how to reset the connector's read point, set Kafka message keys, change topic names, and handle various data types like Bytes, Decimals, and Numerics.
  91. Easy Ways to Generate Test Data in Kafka | ConfluentThis post introduces the Kafka Connect Datagen Connector as a method for generating realistic test data for Kafka topics. It demonstrates how to use the Confluent CLI to start Confluent Platform locally and then configure and deploy the Datagen Connector to produce messages with complex data types, randomized data, and custom schemas. The post also shows how to use the `ksql-datagen` command-line tool for generating data directly to Kafka topics without a connector. It highlights the flexibility of defining custom Avro schemas and configuring output formats like JSON and Avro.
  92. Introducing Confluent Hub: The App Store for Apache Kafka | ConfluentThis post announces the launch of Confluent Hub, an online service and client tool for discovering, reviewing, and downloading extensions for Apache Kafka and Confluent Platform. It highlights connectors, transformations, and converters for Kafka Connect, simplifying their installation and management compared to previous manual methods. The post also introduces the concept of 'Verified Integrations' for technology partners and outlines how developers can contribute components to the hub.
  93. Kafka-Connect-Xenon: Streaming Live with Kafka Connect!This post introduces the Kafka-Connect-Xenon connector, a Confluent-certified Kafka Connect Sink connector developed by Levyx. It details the Xenon data analytics engine, emphasizing its low-latency, high-throughput design for large datasets on SSDs. The post showcases the connector's integration with the Kafka Connect framework and presents performance benchmarks demonstrating an ingest rate of 3.5 million messages per second from Kafka to Xenon on a single server with NVMe SSDs, processing 470GB of data. It also outlines supported data formats (Avro, JSON) and logical types, and discusses the benefits of multi-tenancy and data sharing via Apache Spark.
  94. The Simplest Useful Kafka Connect Data Pipeline in the World – Part 2This post details the configuration and deployment of the Elasticsearch sink connector for Kafka Connect, demonstrating how to stream data from a Kafka topic (populated by a JDBC source connector in a previous article) into an Elasticsearch index. It highlights the use of Avro as a data serialization format, the configuration of the connector via a JSON file and the Confluent CLI, and the resulting data mapping and indexing in Elasticsearch. It also touches upon the use of topic/partition/offset as the document ID for exactly-once delivery and the preservation of data types in Elasticsearch.
  95. The Simplest Useful Kafka Connect Data Pipeline in the World – Part 1This post introduces a foundational Kafka Connect data pipeline, demonstrating the integration of a MySQL database with Apache Kafka using the JDBC source connector. It details the configuration steps for the connector, including setting up the database table and inserting data. The post also shows how to stream this data to both a flat file and Elasticsearch, highlighting the ease of use and scalability of Kafka Connect for building data integration pipelines without custom coding. It emphasizes the use of distributed mode for future scalability and the importance of the `update_ts` column for capturing changes.
  96. Kinetica Releases Confluent Certified Connector for Apache Kafka®This post announces the release and certification of a new Kafka Connector for Kinetica, a GPU-accelerated, in-memory analytics database. The connector provides two main components: KineticaSourceConnector for streaming data from Kinetica to Kafka, and KineticaSinkConnector for writing data from Kafka to Kinetica. It details the data format (Kafka Connect 'Struct') and the deployment mechanism via Kafka Connect API. The source code is made available, and use cases for real-time analytics and machine learning are highlighted.
  97. Streaming Data from Oracle using Oracle GoldenGate and the Connect API in Kafka | ConfluentThis post details the setup and demonstration of using Oracle GoldenGate as a source connector and Elasticsearch as a sink connector within the Kafka Connect API. It highlights the advantages of Oracle GoldenGate's log-based Change Data Capture (CDC) over periodic polling for lower latency and reduced database load. The configuration steps for Oracle GoldenGate's Replicat process and the Kafka Connect handler are provided, along with notes on environment setup and necessary component installations.
  98. The Connect API in Kafka Cassandra Sink: The Perfect Match | ConfluentThis post introduces the Kafka Cassandra Sink connector, a DataStax Certified Connector developed by DataMountaineer. It details how the connector converts Kafka Connect SinkRecords to JSON and uses a fail-fast thread pool for asynchronous insertion into Cassandra. Key features include field selection, topic-to-table routing, error policies (RetriableException), and SSL support. It highlights the use of Kafka Connect Query Language (KCQL) for configuration and demonstrates a use case with Confluent Platform, Cassandra Sink, and Cassandra for a client's streaming analytics platform. The post also provides configuration examples and instructions for setting up Kafka Connect, Cassandra, and submitting the connector configuration.
  99. Build and monitor Kafka pipelines with Confluent Control Center | ConfluentThis post introduces Confluent Control Center, highlighting its capabilities for managing Kafka Connectors and monitoring data streams. It details how Control Center's Kafka Connect configurator automatically renders forms for connectors based on the Kafka Connect API, simplifying pipeline creation. It also explains Stream Monitoring, which provides end-to-end metrics to verify message delivery, identify lost or duplicated messages, and drill down into specific clients, topics, and partitions. The monitoring works with custom Kafka clients, Kafka Connect, and Kafka Streams.
  100. Kafka Connect Sink for PostgreSQL from JustOne Database | ConfluentThis post introduces a new open-source Kafka sink connector for PostgreSQL-compatible databases. It details the use case of creating a relational data lake by streaming enterprise messages into a PostgreSQL database for agile analytics. Performance benchmarks show high ingestion rates (over 200,000 messages/second) into unindexed tables. The post explains the message-to-row conversion mechanism using JSON parsing paths and properties like `db.columns` and `db.json.parse`. It also covers database connection properties, target table definition, and delivery semantics (Fastest, Guaranteed, Synchronized), with Synchronized being the default for exactly-once delivery. Buffer size configuration and Java heap options are also discussed.
  101. Announcing Kafka Connect: Building large-scale low-latency data pipelines | ConfluentThis post announces the introduction of Kafka Connect, a framework for building and managing stream data pipelines using Apache Kafka. It addresses the challenges of data integration by abstracting common problems such as schema management, fault tolerance, parallelism, latency, delivery semantics, and operations/monitoring. Kafka Connect supports Source and Sink connectors, leverages Kafka's parallelism model and offset management for scalability and fault tolerance, and aims to foster a rich ecosystem of connectors.