The Simplest Useful Kafka Connect Data Pipeline in the World – Part 2

This post details the configuration and deployment of the Elasticsearch sink connector for Kafka Connect, demonstrating how to stream data from a Kafka topic (populated by a JDBC source connector in a previous article) into an Elasticsearch index. It highlights the use of Avro as a data serialization format, the configuration of the connector via a JSON file and the Confluent CLI, and the resulting data mapping and indexing in Elasticsearch. It also touches upon the use of topic/partition/offset as the document ID for exactly-once delivery and the preservation of data types in Elasticsearch.