Kafka Connect Single Message Transforms

11/7/2019

This post introduces and details Kafka Connect's Single Message Transforms (SMTs) capability. It explains the concept, provides configuration examples for built-in transforms like RegexRouter and ValueToKey, demonstrates chaining multiple transforms, and offers a deep dive into writing custom SMTs with a Java code example for inserting UUIDs. It also defines the scope and limitations of SMTs, distinguishing them from stream processing tasks.

10/23/2019

This post introduces the Kafka Connect API as a way to build custom connectors for streaming data into and out of Kafka. It outlines the basic architecture of Kafka Connect, including connectors, tasks, and workers, and explains the concepts of standalone and distributed modes. The post then details a four-step process for developing a custom connector: defining configuration properties using AbstractConfig, implementing taskConfigs to distribute configurations to tasks, implementing SourceTask for the core logic, and implementing SourceConnector to define the connector's lifecycle. It also mentions the use of Single Message Transforms (SMTs) and integration with Confluent Schema Registry.

3/13/2019

Introduces and details Kafka Connect's error handling capabilities, specifically focusing on the 'fail fast' mode, silently ignoring errors, and implementing a Dead Letter Queue (DLQ) for problematic messages. It provides configuration examples for `errors.tolerance`, `errors.deadletterqueue.topic.name`, and `errors.deadletterqueue.context.headers.enable`. It also explains how to inspect DLQ messages and mentions JMX metrics for monitoring.

11/14/2018

This post provides a deep dive into Kafka Connect converters and serialization, explaining how Kafka messages are just bytes and the importance of choosing a serialization format (JSON, Avro, Protobuf, String delimited). It details how converters handle serialization and deserialization, the configuration of converters (including key.converter and value.converter), and common errors encountered when using JSON and Avro converters. It also covers troubleshooting tips and internal converters.

9/7/2017

This post introduces and demonstrates the use of Single Message Transforms (SMTs) in Kafka Connect. It covers setting keys for data from the JDBC Source Connector, adding metadata for data lineage (topic name, source details), field masking and blacklisting, and message routing using regular expressions and timestamps. It also provides configuration examples and discusses potential issues with Elasticsearch integration.

2/18/2016

This post introduces Kafka Connect, a new feature in Apache Kafka 0.9+ designed to simplify the building and management of stream data pipelines. It addresses common challenges in data integration, such as schema management, fault tolerance, parallelism, latency, delivery semantics, and operations. Kafka Connect provides a framework for Sources (importing data into Kafka) and Sinks (exporting data from Kafka), abstracting away these complexities. It leverages Kafka's built-in capabilities for parallelism, offset management, and group management to enable scalable and resilient data integration. The post highlights the framework's design for large-scale data integration, its parallelism model based on stream partitions, and its integration with Kafka's offset and group management mechanisms. It also discusses delivery guarantees (at-least-once, at-most-once) and its agnostic approach to process deployment and resource management.