Kafka Client-Broker Compatibility

4 posts · Nov 2014 – Apr 2019

What this trail covers

This feature thread tracks the evolution of Apache Kafka's reliability and operability, focusing on bug fixes and enhancements to core components like replication, MirrorMaker, logging, log compaction, connection handling, metadata management, and the request purgatory. It emphasizes the importance of correctness and stability over new features, with a significant number of bugs fixed in the 0.8.2 release. The thread also covers the introduction of a new distributed testing framework to ensure compatibility and the new producer client.

All Confluent trails

Apr 201912 Programming Languages Walk into a Kafka Cluster… | ConfluentThis post compiles 12 tech tips, each showcasing a different programming language (C/C++, Go, Java, .NET, Python, Scala, Erlang, Groovy, Haskell, Kotlin, Lua, Node.js, OCaml, PHP, Ruby, Rust, Tcl, Swift, Clojure) with examples of Kafka producer and consumer implementations. It highlights the role of librdkafka as a common C client library and provides links to GitHub examples for each language, demonstrating how to connect to Confluent Cloud.
Jul 2017Upgrading Apache Kafka Clients Just Got EasierThis post introduces bidirectional client compatibility for Apache Kafka, enabled by KIP-35 and KIP-97. It explains how newer clients can communicate with older brokers and vice-versa, simplifying upgrades. The technical implementation relies on API versioning and the ApiVersionsRequest mechanism, where clients query brokers for supported API versions. This allows for independent client and broker upgrades, reducing downtime and operational overhead. The post also highlights the `api.version.request` configuration for source-available clients and introduces the `kafka-broker-api-versions.sh` command-line tool for inspecting broker API support.
Feb 2017The First Annual State of Apache Kafka® Client Use Survey | ConfluentThis post presents the findings of the first annual State of Apache Kafka® Client Use Survey. It details language adoption trends (Java, Scala, Python, Clojure, Go, NodeJS, C#, Pony, Cobol) and user priorities for Kafka clients, emphasizing reliability and performance as critical factors. It also offers advice for Kafka client developers based on survey feedback, focusing on stability, consistent APIs, simplicity, and documentation.
Dec 2016Log Compaction | Highlights in the Apache Kafka and Stream Processing Community | December 2016 | ConfluentThis post highlights KIP-97, which proposes an improved Kafka Client RPC Compatibility Policy. This KIP will enable newer versions of Java Kafka clients to communicate with older broker versions, a capability that was previously only supported in the reverse direction. This decoupling of client and server upgrades is expected to simplify the process of upgrading Kafka clusters.
Feb 2016Announcing Apache Kafka 0.9.0.1 and Confluent Platform 2.0.1 | ConfluentThis post announces the release of Apache Kafka 0.9.0.1 and Confluent Platform 2.0.1, which includes fixes for 68 issues, 45 of which are bug fixes. Notable bug fixes include improvements to the new Java consumer (addressing rebalancing issues, offset delivery, and fetch_min_bytes documentation), compatibility fixes (handling null primitives, JSON versions, and reserved broker IDs), security fixes (ticket renewal thread, ACL space handling, SASL packet OOM, join group request OOM, and SSL client auth for SASL_SSL), performance/memory usage fixes (fetch.wait.max.ms, CPU intensive consumer poll, log cleaner configuration, and sensor leaks), and topic deletion fixes (undeletable topics marked for delete).
Advertisement
Jan 2016290 Reasons to Upgrade to Apache Kafka 0.9.0.0 | ConfluentThis post details numerous bug fixes and improvements in Apache Kafka 0.9.0.0, specifically addressing issues in replication (race conditions, min.insync.replica, lag configuration), MirrorMaker (message loss prevention), application logging (cleanup and manageability), log compaction (support for compressed messages), connection leaks, broker metadata caching, request purgatory performance, and producer timeouts. It highlights the introduction of a new distributed testing framework and over 100 new test scenarios to improve quality and confidence in future releases.
Dec 2015Log Compaction | Highlights in the Kafka and Stream Processing Community | January 2016 | ConfluentThis post highlights the release of Kafka 0.9 and Confluent Platform 2.0, which includes improvements to log compaction. It also mentions KIP-40 for group management protocol, KIP-32 for adding timestamps to messages, and the introduction of Kafka Connect for integrating Kafka with external data systems. The post also announces Kafka Summit and congratulates a new Apache Kafka committer.
Jun 2015Hands-Free Kafka Replication: A Lesson in Operational Simplicity | ConfluentThis post introduces a significant improvement to Kafka's replication protocol by replacing the message-count-based lag detection (replica.lag.max.messages) with a time-based approach (replica.lag.time.max.ms). This change addresses the issue of unnecessary ISR churn caused by varying message batch sizes, which previously led to false alerts and operational overhead. The new mechanism ensures that replicas are only removed from the ISR if they consistently lag behind the leader for a configurable duration, improving stability and predictability.
Apr 2015Compatibility Testing For Apache Kafka | ConfluentThis post announces a project to develop high-quality tests for certifying API and protocol compatibility between Kafka versions and distributions. This initiative aims to address the challenges of building reliable distributed systems and ensure that companies can build real-time data flow and processing around Kafka without fear of incompatibilities.
Nov 2014What's coming in Apache Kafka 0.8.2 | ConfluentThis post announces the forthcoming Apache Kafka 0.8.2 release, highlighting significant new features and operational improvements. Key additions include a completely rewritten JVM producer client with unified sync/async behavior and improved batching, a working delete topic feature, and native offset storage functionality using a compacted Kafka topic (__consumer_offsets) to improve scalability and reduce ZooKeeper bottleneck. Operational improvements include automated leader rebalancing, controlled shutdown for brokers, stronger durability guarantees with options for disabling unclean leader election and specifying min.isr, and connection quotas (max.connections.per.ip) to prevent broker unresponsiveness. The post also mentions future plans for security and a new consumer client.