Sending Events to OpenSearch with the New Confluent Sink Connector
May 21, 2024Ahmed Saef Zamzam

Summary
Introduces a new fully managed OpenSearch Sink Connector for Confluent Cloud, simplifying the process of streaming data from Kafka topics to OpenSearch indexes. Details the connector's architecture, configuration options (1:1, M:1, 1:M, N:M topic-to-index mappings), scalability through connector tasks, and cost considerations. Addresses limitations and workarounds for private networking with OpenSearch (PrivateLink, Transit Gateway, VPC peering) and notes the current lack of support for batch inserts and OpenSearch Serverless authentication, with ongoing work to address these.