OpenSearch Ingestion Integration

6/26/2024

This post details the technical implementation of integrating AWS OpenSearch Ingestion with Confluent Cloud. It explains the networking challenges and solutions, specifically how OpenSearch Ingestion uses xENIs to connect to private Confluent Cloud clusters via PrivateLink. It also covers authentication mechanisms (SASL/PLAIN with API keys for Confluent, IAM roles for OpenSearch) and scaling via OpenSearch Capacity Units (OCUs). A practical guide is provided for creating an OpenSearch Ingestion pipeline using a Confluent blueprint, including configuration parameters for topics, bootstrap servers, endpoints, and secrets management.

9/26/2019

This post details the integration of Apache Kafka with Rockset for real-time analytics and monitoring dashboards. It explains how to use the Rockset Kafka Connector to ingest data from Kafka topics into Rockset collections, enabling interactive SQL queries on raw JSON data without upfront ETL. The post demonstrates joining Kafka data with external datasets (e.g., company information) and creating real-time dashboards using BI tools like Redash. It highlights Kafka's role as an event streaming platform for critical infrastructures and its ability to decouple producers and consumers.

6/18/2019

This post details building a scalable search architecture using Kafka Connect with JDBC Source and Elasticsearch Sink connectors. It explains the challenges of using RDBMS for search, introduces Kafka as a middleware for scalability and flexibility, and outlines two Kafka Connect-based approaches for indexing pipelines. The JDBC connector section covers its modes of operation (incrementing, timestamp, combination, custom query, bulk import) and schema evolution with Schema Registry. The Elasticsearch connector section provides a basic configuration for indexing data from Kafka to Elasticsearch. It also discusses the importance of considering query performance for the JDBC connector and mapping strategies for Elasticsearch.

3/22/2018

This post details how to enrich CSV events in real-time by joining them with customer data from a MySQL RDBMS using Kafka Connect and ksqlDB. It covers setting up customer data in MySQL, using Mockaroo for data generation, and configuring Kafka Connect to ingest data from MySQL into a Kafka topic. The post then demonstrates how ksqlDB can be used to perform a stream-table join between the order events (CSV) and customer data (from MySQL) to enrich the order events with customer attributes. The resulting enriched stream is then sent to AWS S3 for analysis.