TechPrufer
Get started
Confluent
Confluent logoConfluent · Feature trail

HTTP and REST API Integration with Kafka

10 posts · Mar 2015 – Jun 2025

What this trail covers

Confluent Cloud continues to evolve its integration capabilities for HTTP and REST APIs with Kafka, focusing on enhancing the HTTP Source V2 and HTTP Sink V2 connectors. These connectors now support configuring multiple APIs with the same hostname and authorization, seamless integration with OpenAPI Specification 3.0, API testing during configuration, offset management, and client-side field-level encryption. The HTTP Sink V2 also allows specifying Kafka record fields in the URL, parameters, and headers. This post details the creation of a node.js wrapper for the Kafka REST Proxy, demonstrating how to build applications by exposing Kafka functionality through HTTP. It highlights improvements in the produce API for handling various message formats and serialization, and a more native Node.js-like consumer API using event emitters. An example application is provided that streams Twitter data into Kafka and processes trending topics.

All Confluent trails
  1. Integrate Kafka with AWS EventBridge Using ConfluentThis post introduces the Amazon EventBridge Sink Connector for Confluent Platform, enabling bidirectional integration between Kafka and AWS EventBridge. It details use cases in financial services, manufacturing, and healthcare, and provides step-by-step instructions for setting up the integration on local Confluent Platform and AWS. Best practices for production include aligning service quotas, configuring dead-letter queues, enforcing least-privilege IAM, handling large events with S3 offloading, and leveraging logging for debugging. The connector allows Kafka events to trigger AWS Lambda functions, SNS notifications, SQS queues, and update data lakes, extending the reach of event streams into the AWS ecosystem.
  2. Provisioned Mode for Lambda's Kafka ESM: New Controls and AutoscalingIntroduces Provisioned Mode for AWS Lambda's Kafka Event Source Mapping (ESM), offering fine-grained control over throughput via configurable minimum and maximum event pollers. This mode simplifies networking by eliminating the need for PrivateLink or NAT Gateway, and provides responsive auto-scaling to handle traffic spikes. Performance comparisons demonstrate significant improvements in message backlog processing times compared to the default On-Demand mode.
  3. Connect Your HTTP APIs to Confluent with Fully Managed ConnectorsThis post introduces the HTTP Source V2 and HTTP Sink V2 connectors, detailing their capabilities for integrating with HTTP APIs. It highlights new features such as multi-API configuration, OpenAPI 3.0 support, API testing, offset management, and client-side field-level encryption. The post provides a concrete example of using these connectors to ingest data from Stripe, process it with Flink and Azure OpenAI for vector embeddings, and then sink it to Pinecone. It also outlines future roadmap items for both connectors.
  4. Building Event-Driven Architectures: Top 5 Best PracticesThis post details two patterns for integrating Confluent Cloud with AWS Lambda for event-driven architectures: the fully managed AWS Lambda Sink Connector and native Event Source Mapping (ESM). It highlights the benefits and considerations of each, including throughput, latency, auto-scaling, and error handling. The post also outlines five best practices: batching calls to Lambda, implementing idempotent Lambda functions, handling errors to avoid duplicates and stalled partitions, and keeping long-lived connections to Kafka outside the handler for efficiency. It emphasizes the trade-offs between the two integration patterns for different use cases.
  5. Two Ways to Build Event-Driven Serverless ApplicationsThis post details two patterns for integrating AWS Lambda with Kafka: the Confluent Lambda Sink Connector and AWS Lambda Event Source Mapping (ESM). The Sink Connector offers synchronous/asynchronous invocation and DLQ support, suitable for fire-and-forget scenarios or ordered processing with tasks. ESM provides automatic polling and scaling based on consumer offset lag, ideal for auto-scaling and ordered processing within a consumer group. Use cases are provided for each pattern: healthcare EHR integration with ESM for ordered processing, and email marketing with the Sink Connector for asynchronous, high-throughput scenarios.
  7. Serverless, Secure Data Streaming with Confluent Cloud and AWS LambdaThis post details the integration of AWS Lambda with Confluent Cloud for serverless data streaming. It highlights the use of Lambda's SASL/PLAIN authentication support to trigger Lambda functions directly from Kafka topics, enabling data ingestion into Amazon DynamoDB. The post provides a step-by-step guide for setting up Confluent Cloud, creating Kafka clusters and topics, and deploying AWS resources via CloudFormation. It also covers configuring the Datagen Source connector to generate sample data for the 'orders' topic.
  8. Event Streaming Across Networks and Corporate Firewalls Using PubNub and Confluent PlatformThis post introduces the PubNub Bridge for Apache Kafka with Confluent Platform, a beta product that enables bidirectional communication between mobile/web applications and Kafka topics. It addresses the complexities of building gateway APIs for Kafka by providing a unified, secure, and scalable solution that bypasses the need for complex IT projects, firewall configurations, and direct API interfaces. The bridge allows direct connection of mobile and web apps to Kafka topics, facilitating use cases like real-time alerts, notifications, and actions. It also details a practical demonstration using Docker to set up Kafka, ZooKeeper, and the bridge itself, highlighting the configuration of API keys and channel settings.
  9. Kafka Tools that Boost Developer ProductivityThis post details the usage of `kafkacat` for producing and consuming messages, including handling keys and Avro data. It also explains how to use the Confluent REST Proxy to produce Avro data and consume messages via HTTP requests, demonstrating the creation and management of consumers. Finally, it highlights the utility of `jq` for processing JSON output from tools like the REST Proxy, showing how to extract specific data fields.
  10. Confluent REST Proxy 6.0: Putting Apache Kafka to RESTConfluent REST Proxy 6.0 introduces two major improvements: 1. Expanded administrative functionality via RESTful HTTP APIs for resources including ACLs, brokers, topics, and consumer groups. 2. Integration of REST Proxy APIs directly into Confluent Server, eliminating the need for separate REST Proxy nodes and simplifying operational management. The post provides examples of using these new APIs for cluster information retrieval, topic creation and configuration updates, broker management, and ACL management.
  11. Real-Time Serverless Ingestion, Streaming and Analytics Using Confluent and AWSThis post details a serverless architecture for ingesting, streaming, and analyzing data using Confluent Cloud and AWS services. It provides a step-by-step guide for setting up AWS resources (Lambda, Timestream, QuickSight) via CloudFormation, creating a Confluent Cloud Kafka cluster and topic, and configuring the AWS Lambda Sink Connector to stream data from Kafka to AWS Lambda, which then ingests it into Amazon Timestream. The post highlights the benefits of a serverless approach, including elastic scaling, usage-based billing, and reduced operational burden, and demonstrates a practical use case of streaming stock data for real-time analysis.
  12. HTTP and REST APIs with Apache Kafka - Use Cases and ArchitecturesThis post details the use cases and architectures of REST APIs and the Confluent REST Proxy for interacting with Apache Kafka. It categorizes use cases into management plane (cluster configuration, CI/CD) and data plane (mobile apps, legacy integration, API gateways, other programming languages). It highlights the benefits of REST APIs for Kafka integration, including familiarity, security (easier firewall traversal), and alignment with Domain-Driven Design. The post differentiates between the data plane (decoupled REST Proxy instances for scalability) and management plane (embedded REST APIs in Confluent Server/Cloud). It also emphasizes the role of Confluent Schema Registry for data governance and schema enforcement with REST services, supporting Avro, Protobuf, and JSON Schema.
  13. Push Real-Time Data to the Web with the Kafka Connect HTTP SinkThis post introduces the Kafka Connect HTTP Sink Connector and demonstrates its use in pushing real-time data from Kafka to web services. It provides a step-by-step guide for setting up Confluent Platform locally, installing and configuring the HTTP Sink Connector, and running a Node.js web service to receive and forward data as Server-Sent Events (SSE) to a frontend. The post also includes instructions for testing the end-to-end pipeline by producing messages to Kafka and observing them in the web browser.
  14. Getting started with Kafka in node.js with the Confluent REST Proxy | ConfluentThis post details the creation of a node.js wrapper library for the Confluent REST Proxy. It showcases how to expose Kafka functionality (topics, partitions, consumers, brokers) via HTTP. Key technical contributions include a flexible `produce` API that handles various message formats (single/batch, binary/Avro, optional fields) and a consumer API that leverages Node.js's EventEmitter for a stream-like interface. An example application is provided that streams Twitter data into Kafka and processes trending topics, demonstrating the practical use of the wrapper.
  15. A Comprehensive REST Proxy for Kafka | ConfluentThis post introduces the Kafka REST Proxy, an HTTP-based proxy for Kafka clusters. It details its features, including producing/consuming messages, accessing cluster metadata, and first-class support for Avro with Schema Registry integration. The proxy is designed for scalability and can be deployed in clusters behind load balancers. It addresses the need for languages beyond those with native Kafka clients and offers functionality beyond traditional clients for cluster management. The post provides cURL examples for producing and consuming Avro messages and explains the proxy's architecture, which wraps Java libraries, uses JSON with flexible embedded data, and employs stateful consumers that are distributed and load-balanced. The rationale for building the REST Proxy is explained, highlighting the complexity of building feature-complete, high-performance Kafka clients, especially for the flexible consumer group model.