The Simplest Useful Kafka Connect Data Pipeline in the World – Part 3

This post details the practical application of Kafka Connect Single Message Transforms (SMTs) for common data pipeline tasks. It demonstrates how to use SMTs to set message keys from database columns (e.g., JDBC source), embed topic names and custom source details for data lineage, mask or drop sensitive fields (e.g., MaskField, ReplaceField), and route messages to different topics using regular expressions (RegexRouter) or timestamps (TimestampRouter). It also provides configuration examples and workarounds for potential issues like null keys in Elasticsearch sinks.