HTTP and REST API Integration with Kafka

8/2/2023

This post details two patterns for integrating AWS Lambda with Confluent Cloud: the Confluent Lambda Sink Connector and AWS Lambda's event source mapping (ESM) for Kafka. The sink connector supports synchronous (ordered processing, task parameter) and asynchronous (at-most-once, DLQ) invocations. ESM polls Kafka topics and auto-scales consumers based on offset lag, suitable for auto-scaling. Both support ordered processing. Use cases include healthcare EHR integration (ESM) and email marketing (Sink Connector).

1/21/2021

This post introduces the PubNub Bridge for Apache Kafka with Confluent Platform, a beta product that enables bidirectional communication between mobile/web applications and Kafka topics. It addresses the complexities of building gateway APIs for Kafka by providing a unified, secure, and scalable solution that bypasses the need for complex IT projects, firewall configurations, and direct API interfaces. The bridge allows direct connection of mobile and web apps to Kafka topics, facilitating use cases like real-time alerts, notifications, and actions. It also details a practical demonstration using Docker to set up Kafka, ZooKeeper, and the bridge itself, highlighting the configuration of API keys and channel settings.

1/20/2021

This post details the usage of `kafkacat` for producing and consuming messages, including handling keys and Avro data. It also explains how to use the Confluent REST Proxy to produce Avro data and consume messages via HTTP requests, demonstrating the creation and management of consumers. Finally, it highlights the utility of `jq` for processing JSON output from tools like the REST Proxy, showing how to extract specific data fields.

1/11/2021

This post details the enhancements to Confluent REST Proxy in Confluent Platform 6.0. It introduces the ability to administer Apache Kafka via RESTful HTTP APIs for resources like ACLs, brokers, clusters, topics, partitions, and consumer groups. It also highlights the integration of REST Proxy APIs directly into Confluent Server, removing the need for a separate REST Proxy deployment. Examples are provided for cluster and topic management, including creating topics with configurations, resetting topic configurations, and managing ACLs. It also touches on managing brokers and their configurations.

12/1/2020

This post details the integration of Confluent Cloud with AWS serverless services, specifically AWS Lambda and Amazon Timestream, to build a real-time serverless ingestion, streaming, and analytics pipeline. It provides a step-by-step guide on using the AWS Lambda Sink Connector to stream data from Confluent Cloud to Amazon Timestream. The post also outlines the prerequisites, creation of AWS resources via CloudFormation, Confluent Cloud cluster and topic setup, API key generation, and the configuration of the AWS Lambda Sink Connector. It highlights the serverless nature of the architecture, enabling users to focus on application development without managing infrastructure.

11/18/2020

Introduces the Confluent REST Proxy as a means to interact with Kafka via HTTP/REST. Details management plane use cases (cluster configuration, CI/CD) and data plane use cases (mobile apps, legacy systems, API gateways, multi-language support). Explains the architecture of the REST Proxy in Confluent Platform and Cloud, its integration with Schema Registry for data governance, and provides tutorials for getting started.

4/20/2020

This post details building an end-to-end event-driven alerting system by using the Kafka Connect HTTP Sink Connector to push real-time data from Kafka to web services, which then forward it as Server-Sent Events (SSE) to an HTML5 client. It covers setting up Confluent Platform locally, configuring the HTTP Sink Connector, and developing a Node.js service to receive data and broadcast it via SSE.

7/23/2015

This post details the creation of a node.js wrapper library for the Confluent REST Proxy. It explains the design choices for the wrapper, focusing on mirroring the REST API and providing a user-friendly interface for producing messages (handling various formats, serialization, and batching) and consuming messages (using an EventEmitter-like API). It also presents a practical application example: a Twitter trending topics aggregator that demonstrates streaming data from Twitter to Kafka using the wrapper, processing it with Avro schemas, and exposing the results.

3/25/2015

Introduces the Confluent REST Proxy as an HTTP-based interface for Apache Kafka. Details its features, including producing/consuming messages, accessing cluster metadata, and integration with Schema Registry. Explains the technical design choices: wrapping Java libraries, using JSON with flexible embedded data (Avro support via JSON encoding and binary serialization), stateful consumers, and distributed/load-balanced deployment. Provides cURL examples for producing and consuming Avro messages and explains the rationale for building a REST proxy due to the complexity of Kafka client development, especially the consumer group model.