Kafka Connect Single Message Transforms

6 posts · Feb 2016 – Nov 2019

What this trail covers

This feature thread tracks the evolution of Kafka Connect's Single Message Transforms (SMTs), enabling in-flight message manipulation without custom code. This includes setting message keys, adding metadata for data lineage, masking or dropping fields, and routing messages to different topics based on regex patterns or timestamps. SMTs enhance data pipelines by allowing for pre-processing, enrichment, and flexible routing before data reaches its final destination or is processed by downstream systems like Elasticsearch.

All Confluent trails

Nov 2019Single Message Transforms in Kafka ConnectThis post introduces and details Kafka Connect's Single Message Transforms (SMTs) capability. It explains the concept, provides configuration examples for built-in transforms like RegexRouter and ValueToKey, demonstrates chaining multiple transforms, and offers a deep dive into writing custom SMTs with a Java code example for inserting UUIDs. It also defines the scope and limitations of SMTs, distinguishing them from stream processing tasks.
Oct 20194 Steps to Creating Apache Kafka Connectors with the Kafka Connect APIThis post introduces the Kafka Connect API as a way to build custom connectors for streaming data into and out of Kafka. It outlines the basic architecture of Kafka Connect, including connectors, tasks, and workers, and explains the concepts of standalone and distributed modes. The post then details a four-step process for developing a custom connector: defining configuration properties using AbstractConfig, implementing taskConfigs to distribute configurations to tasks, implementing SourceTask for the core logic, and implementing SourceConnector to define the connector's lifecycle. It also mentions the use of Single Message Transforms (SMTs) and integration with Confluent Schema Registry.
Jun 2019How to Connect KSQL to Confluent Cloud using Kubernetes with Helm | ConfluentThis post details the specific Helm chart configurations required to connect Confluent Platform components (KSQL, Schema Registry, Kafka Connect) to Confluent Cloud. It provides guidance on gathering client configuration, updating `values.yaml` files with `configurationOverrides` for connection parameters, and installing these components into a Kubernetes cluster using Helm. It also outlines a concrete example of setting up a KSQL tutorial environment by deploying Schema Registry, KSQL, and Kafka Connect with the Datagen connector into Kubernetes.
May 2019Dawn of Kafka DevOps: Managing Multi-Cluster Kafka Connect and KSQL with Control Center | ConfluentThis post introduces the use of Confluent Control Center for managing multiple Kafka Connect and KSQL clusters. It demonstrates how to use Control Center to select different clusters for various tasks, such as ingesting data with Kafka Connect, transforming data with KSQL, streaming data to a database with Kafka Connect, and performing analytics with KSQL. The post highlights the benefits of multi-cluster deployments for resource isolation, environment separation, and application separation.
May 2019Dawn of Kafka DevOps: Managing Kafka Clusters at Scale with Control Center | ConfluentConfluent Platform 5.2 enhances Confluent Control Center's capabilities for managing large Kafka deployments. It introduces the ability to visualize differences in broker configurations and dynamically modify them without requiring rolling restarts, addressing challenges endemic to large distributed systems. Control Center's scalability has also been significantly improved, now comfortably handling around 120,000 individual partition replicas (approximately 40,000 partitions with a replication factor of three), covering over 90% of existing customer workloads.
Advertisement
Mar 2019Kafka Connect Deep Dive – Error Handling and Dead Letter Queues | ConfluentIntroduces and details Kafka Connect's error handling capabilities, specifically focusing on the 'fail fast' mode, silently ignoring errors, and implementing a Dead Letter Queue (DLQ) for problematic messages. It provides configuration examples for `errors.tolerance`, `errors.deadletterqueue.topic.name`, and `errors.deadletterqueue.context.headers.enable`. It also explains how to inspect DLQ messages and mentions JMX metrics for monitoring.
Nov 2018Kafka Connect Deep Dive – Converters and Serialization Explained | ConfluentThis post provides a deep dive into Kafka Connect converters and serialization, explaining how Kafka messages are just bytes and the importance of choosing a serialization format (JSON, Avro, Protobuf, String delimited). It details how converters handle serialization and deserialization, the configuration of converters (including key.converter and value.converter), and common errors encountered when using JSON and Avro converters. It also covers troubleshooting tips and internal converters.
Sep 2017The Simplest Useful Kafka Connect Data Pipeline in the World – Part 3This post details the practical application of Kafka Connect Single Message Transforms (SMTs) for common data pipeline tasks. It demonstrates how to use SMTs to set message keys from database columns (e.g., JDBC source), embed topic names and custom source details for data lineage, mask or drop sensitive fields (e.g., MaskField, ReplaceField), and route messages to different topics using regular expressions (RegexRouter) or timestamps (TimestampRouter). It also provides configuration examples and workarounds for potential issues like null keys in Elasticsearch sinks.
Feb 2016Announcing Kafka Connect: Building large-scale low-latency data pipelines | ConfluentThis post introduces Kafka Connect, a new feature in Apache Kafka 0.9+ designed to simplify the building and management of stream data pipelines. It addresses common challenges in data integration, such as schema management, fault tolerance, parallelism, latency, delivery semantics, and operations. Kafka Connect provides a framework for Sources (importing data into Kafka) and Sinks (exporting data from Kafka), abstracting away these complexities. It leverages Kafka's built-in capabilities for parallelism, offset management, and group management to enable scalable and resilient data integration. The post highlights the framework's design for large-scale data integration, its parallelism model based on stream partitions, and its integration with Kafka's offset and group management mechanisms. It also discusses delivery guarantees (at-least-once, at-most-once) and its agnostic approach to process deployment and resource management.