TechPrufer
Get started
Confluent
Confluent logoConfluent · Feature trail

OpenSearch Ingestion Integration

4 posts · Mar 2018 – Jun 2024

What this trail covers

Confluent Cloud continues to evolve its integration capabilities for OpenSearch, focusing on simplifying data ingestion and analytics. This iteration introduces support for Amazon OpenSearch Ingestion as a source, enabling seamless, private connectivity between Confluent Cloud and OpenSearch domains/collections. This complements the existing managed OpenSearch Sink connector, providing a comprehensive solution for streaming data to OpenSearch.

All Confluent trails
  1. AWS OpenSearch Ingestion Adds Confluent Blueprint, w/Pipeline Ex.This post details the technical integration between Amazon OpenSearch Ingestion and Confluent Cloud, specifically focusing on private connectivity using AWS PrivateLink and Elastic Network Interfaces (xENIs). It explains the networking challenges of unidirectional traffic in PrivateLink and how xENIs enable OpenSearch Ingestion to connect to third-party private data sources like Confluent Cloud. The post also covers authentication mechanisms (SASL/PLAIN with API keys for Confluent, IAM roles for OpenSearch) and scaling considerations based on OpenSearch Capacity Units (OCUs) and Kafka partitions. A step-by-step guide for configuring the OpenSearch Ingestion pipeline using a Confluent blueprint is provided, including details on secrets management and network configuration.
  2. Sending Events to OpenSearch with the New Confluent Sink ConnectorThis post announces and details the new fully managed OpenSearch Sink Connector for Confluent Cloud. It explains how the connector works, its configuration options (including topic-to-index mapping flexibility), scalability through connector tasks, and pricing. It also highlights considerations such as limitations with Amazon OpenSearch Serverless authentication and private networking, and the current lack of batch insert support. A practical example demonstrates configuring the connector to sink data from multiple topics to different OpenSearch indexes.
  3. Real-Time Analytics and Monitoring Dashboards with Kafka and RocksetThis post details the integration of Apache Kafka with Rockset for real-time analytics and monitoring dashboards. It explains how to use the Rockset Kafka Connector to ingest data from Kafka topics into Rockset collections, enabling interactive SQL queries on raw JSON data without upfront ETL. The post demonstrates joining Kafka data with external datasets (e.g., company information) and creating real-time dashboards using BI tools like Redash. It highlights Kafka's role as an event streaming platform for critical infrastructures and its ability to decouple producers and consumers.
  4. Building a Scalable Search Architecture | ConfluentThis post details building a scalable search architecture using Kafka Connect with JDBC Source and Elasticsearch Sink connectors. It explains the challenges of using RDBMS for search, introduces Kafka as a middleware for scalability and flexibility, and outlines two Kafka Connect-based approaches for indexing pipelines. The JDBC connector section covers its modes of operation (incrementing, timestamp, combination, custom query, bulk import) and schema evolution with Schema Registry. The Elasticsearch connector section provides a basic configuration for indexing data from Kafka to Elasticsearch. It also discusses the importance of considering query performance for the JDBC connector and mapping strategies for Elasticsearch.
  5. KSQL in Action: Enriching CSV Events with Data from RDBMS into AWSThis post details how to enrich CSV events in real-time by joining them with customer data from a MySQL RDBMS using Kafka Connect and ksqlDB. It covers setting up customer data in MySQL, using Mockaroo for data generation, and configuring Kafka Connect to ingest data from MySQL into a Kafka topic. The post then demonstrates how ksqlDB can be used to perform a stream-table join between the order events (CSV) and customer data (from MySQL) to enrich the order events with customer attributes. The resulting enriched stream is then sent to AWS S3 for analysis.