A Comprehensive REST Proxy for Kafka | Confluent

This post introduces the Kafka REST Proxy, an HTTP-based proxy for Kafka clusters. It details its features, including producing/consuming messages, accessing cluster metadata, and first-class support for Avro with Schema Registry integration. The proxy is designed for scalability and can be deployed in clusters behind load balancers. It addresses the need for languages beyond those with native Kafka clients and offers functionality beyond traditional clients for cluster management. The post provides cURL examples for producing and consuming Avro messages and explains the proxy's architecture, which wraps Java libraries, uses JSON with flexible embedded data, and employs stateful consumers that are distributed and load-balanced. The rationale for building the REST Proxy is explained, highlighting the complexity of building feature-complete, high-performance Kafka clients, especially for the flexible consumer group model.