TechPrufer
Get started
Confluent
Confluent logoConfluent · Feature trail

Kafka Producer and Consumer API Enhancements

5 posts · Nov 2014 – Jan 2026

What this trail covers

This feature thread tracks the evolution of Kafka producer and consumer APIs, focusing on reliability, performance, and advanced features like Exactly Once Semantics (EOS). It covers improvements in producer idempotency, transactional guarantees, and the integration of new connectors to enhance data pipeline capabilities. The thread also highlights advancements in consumer group management and offset handling for robust data processing. This post details the internal workings of Kafka producers, including serialization, partitioning, and batching strategies, and how these prepare event data for brokers. Future posts will cover broker handling of producer requests, consumer fetch preparation, and broker-side fetch handling.

All Confluent trails
  1. How a Tier‑1 Bank Tuned Apache Kafka® for p99 Latency for TradingThis post details a Tier-1 bank's successful effort to achieve sub-5ms, p99, end-to-end latency for critical trading pipelines with strict durability requirements. Through architectural discipline, in-depth monitoring, and nuanced configuration of brokers, producers, and consumers, the teams met their goal of 100,000 messages/sec and sustained 1.6 million messages/sec with messages smaller than 5KB. Key technical aspects include using the OpenMessaging Benchmark (OMB) for reproducible testing, focusing on a "lowest baseline first" methodology with a single Kafka partition and four replicas across two data centers, and meticulously instrumenting every stage of the Kafka message path to identify and mitigate tail latency. Specific tuning details are provided for broker metrics (MessagesInPerSec, RequestQueueTimeMs, ISR Health), producer metrics (request-latency-avg, batch-size-avg, linger.ms, batch.size, compression.type, enable.idempotence, acks), and consumer metrics (records-consumed-rate, fetch-latency-avg, max.poll.records, max.poll.interval.ms). Infrastructure upgrades and JVM tuning (Generational ZGC, JDK 21) were also critical.
  2. Kafka Producer Internals: The Producer Request LifecycleThis post details the internal workings of Kafka producers, specifically the request lifecycle from the client to the broker and back. It covers the journey of a produce request through the broker's socket receive buffer, network threads, request queue, I/O threads, page cache, disk segments (.log, .index, .timeindex, .snapshot), purgatory for replication (waiting for acks), response queue, and socket send buffer. It also highlights relevant configurations like `socket.receive.buffer.bytes`, `socket.request.max.bytes`, `num.network.threads`, `queued.max.requests`, `num.io.threads`, `log.flush.interval.ms`, `log.flush.interval.messages`, `log.segment.bytes`, `cleanup.policy`, `default.replication.factor`, `replication.factor`, `replica.fetch.wait.max.ms`, and `num.replica.fetchers`. Key metrics for monitoring include `NetworkProcessorAvgIdlePercent`, `RequestQueueSize`, `RequestQueueTimeMs`, `RequestHandlerAvgIdlePercent`, `LogFlushRateAndTimeMs`, `LocalTimeMs`, `RemoteTimeMs`, `ResponseQueueSize`, `ResponseQueueTimeMS`, `ResponseSendTimeMs`, and `TotalTimeMs`.
  3. Inside the Kafka Black Box—How Producers Prepare Event Data for BrokersThis post details the internal workings of Kafka producers, specifically focusing on the steps involved in preparing event data before it is sent to brokers. It covers serialization (converting data to bytes using serializers and Schema Registry), partitioning (determining the target partition using hash(key) % num_partitions or sticky partitioning, with configurations like `partitioner.class`, `partitioner.ignore.keys`, `partitioner.adaptive.partitioning.enable`, and `partitioner.availability.timeout.ms`), and batching (grouping records destined for the same partition using `batch.size` to improve efficiency and throughput, with considerations for record size and memory usage). The post emphasizes understanding these internal mechanisms for debugging and optimizing producer behavior.
  4. Apache Kafka Message CompressionThis post details the internal workings of Kafka producers, specifically focusing on message compression strategies and their impact on throughput, network traffic, disk utilization, and consumer performance. It outlines available compression algorithms (gzip, snappy, lz4, zstd), configuration options at both topic and producer levels, and best practices for choosing the optimal compression type. It also touches upon the interaction between compression and encryption, and how to inspect compression codecs on brokers.
  5. How to Make Apache Kafka Clients Go Fast(er) on Confluent Cloud | Confluent How to Make Apache Kafka Even FasterThis post provides a curated list of five resources (whitepapers and blog posts) that detail how to optimize Apache Kafka client performance (latency and throughput) on Confluent Cloud. It explains the concepts of 'fast' in terms of latency and throughput, discusses default optimizations in Kafka brokers and clients, and highlights the importance of benchmarking in production environments. The resources cover cloud-native design principles, client configuration best practices (producer and consumer parameters), benchmarking methodologies, and the impact of various factors like partitions, compression, and network on performance.
  7. Apache Kafka and Scala - A Beginner’s TutorialThis post provides a beginner-friendly tutorial on using Apache Kafka with Scala. It covers setting up a Scala project with Kafka clients and Confluent serializers, producing records using `KafkaProducer` with different configurations (batching, single events, transactions), and handling Avro serialization with Schema Registry. It also introduces the `kafka-application4s` example project and its practical use case for content rating.
  8. Top 5 Things Every Apache Kafka Developer Should KnowThis post introduces the sticky partitioner for Kafka producers, which improves batching efficiency by assigning records to the same partition until the batch is full or a timeout occurs, leading to fewer produce requests and reduced latency. It also explains the cooperative rebalancing protocol for consumer groups, which minimizes downtime during rebalances by shifting coordination logic to the broker-side group coordinator. Additionally, it details message delivery and durability guarantees configurable via the `acks` setting and `min.insync.replicas`, and highlights the utility of record headers for passing metadata and debugging information.
  9. Multi-Threaded Messaging with the Apache Kafka ConsumerThis post details the motivation and implementation of a multi-threaded Kafka consumer architecture. It explains the limitations of the single-threaded 'thread per consumer' model, particularly concerning slow processing times and exception handling, and proposes a solution that decouples consumption from processing using a thread pool. Key considerations for maintaining at-least-once delivery semantics and per-partition processing order are discussed, along with the implications of consumer group rebalancing.
  10. Building Systems with Transactions Using Apache Kafka | ConfluentThis post details how to build event-based architectures that retain transactional properties similar to RDBMS by leveraging Apache Kafka, focusing on atomicity, consistency, isolation, and durability. It discusses event modeling, API consistency, and idempotent event processing to achieve transactional guarantees in a distributed system.
  11. 12 Programming Languages Walk into a Kafka Cluster… | ConfluentThis post highlights the expansion of Kafka client libraries to support a wide array of programming languages, including official Confluent clients for C/C++, Go, Java, .NET, Python, and Scala, as well as community-supported clients for languages like Erlang, Groovy, Haskell, Kotlin, Lua, Node.js, OCaml, PHP, Ruby, Rust, Tcl, Swift, and Clojure. It emphasizes the role of librdkafka as a foundational C library for many of these clients and points to example producer and consumer code available on GitHub for Confluent Cloud.
  12. Transactions in Apache Kafka | ConfluentThis post dives deeper into Kafka transactions, explaining their primary use case in read-process-write stream processing applications. It details how transactions enable atomic multi-partition writes and zombie fencing to achieve exactly-once processing. The post outlines transactional semantics, the Java client API for transactions, and the underlying components like the Transaction Coordinator and Transaction Log. It clarifies that while Kafka guarantees committed transactional messages will be delivered and aborted/open ones withheld, it does not provide atomic reads.
  13. Exactly-once Semantics is Possible: Here's How Apache Kafka Does itThis post introduces and explains exactly-once semantics (EOS) in Apache Kafka, detailing how the 0.11 release and Confluent Platform 3.3 enable this capability. It elaborates on the challenges of achieving EOS in distributed systems and breaks down Kafka's solution into two key components: 1. Idempotent Producers: This feature ensures that retried sends of the same message batch only result in a single write to the Kafka log per partition, preventing duplicates due to producer or broker errors. It works by persisting sequence numbers to the replicated log. 2. Transactions API: This allows for atomic writes across multiple partitions, ensuring that either all messages in a batch are visible or none are. It also enables committing consumer offsets within the same transaction for end-to-end EOS. The post also explains how the Kafka Streams API leverages these features with the `processing.guarantee=exactly_once` configuration for exactly-once stream processing, guaranteeing that each received record's processed results are reflected once, even under failures.
  14. Log Compaction – Highlights in the Apache Kafka ® and Stream Processing Community – June 2017 | ConfluentThis post announces the upcoming General Availability of Kafka release 0.11.0.0, highlighting the introduction of Exactly Once Semantics (EOS). EOS provides more stringent idempotent producer semantics with exactly once, in-order delivery per partition, and stronger transactional guarantees with atomic writes across multiple partitions. It also mentions the growing adoption of Kafka Connect API and lists several new community-contributed connectors (AWS S3 sink, Redis source, RabbitMQ source, GitHub source, Amazon Kinesis source, MongoDB sink). The post also references upcoming Kafka Summit events and provides links to notable blogs and presentations discussing Kafka's use in various architectures.
  15. Introduction to Apache Kafka® for Python Programmers | ConfluentThis post introduces the basics of using the confluent-kafka-python client for Apache Kafka. It covers installation, starting a local Kafka cluster with ZooKeeper, producing messages with configuration properties (bootstrap.servers), using callbacks for asynchronous delivery confirmation, and consuming messages with consumer groups, group.id, auto.offset.reset, and the poll loop. It also touches upon handling partition EOF and errors during consumption.
  16. The First Annual State of Apache Kafka® Client Use Survey | ConfluentThis post presents the results of the first annual State of Apache Kafka® Client Use Survey, conducted at the end of 2016. It details community responses regarding Kafka client language adoption and user priorities. Key findings include Java's continued dominance, surprising adoption of Clojure, and the critical importance of reliability and performance for client selection. The survey also provides advice for Kafka client developers, emphasizing stability, consistent APIs, simplicity, and comprehensive documentation.
  17. Finding a Good Apache Kafka Client | ConfluentThis post explains the evolution of Kafka's multi-language support, the design of its binary protocol for backward compatibility, and the challenges of developing high-quality clients. It highlights Confluent's strategy of focusing on the C client (librdkafka) as a foundation for clients in other languages (Python, Go, Node.js, .NET) due to its performance, reliability, and feature completeness. It also mentions the introduction of a REST layer for languages without high-quality clients and the simplification of consumer group management through a new core Kafka protocol.
  18. Log Compaction | Highlights in the Kafka and Stream Processing Community | March 2016 | ConfluentThis post details the implementation of KIP-41 (max records per poll) and KIP-32 (message timestamps) into the new Kafka consumer API. It also covers the merging of KIP-42 (client interceptors) for producers and consumers, and mentions KIP-47 (time-based log deletion) and KIP-48 (delegation tokens) as new KIPs under discussion. Additionally, it announces the release of Apache Kafka 0.9.0.1 and Confluent Platform 2.0.1, and the upcoming Kafka 0.10.0 release which will include Kafka Streams.
  19. What's coming in Apache Kafka 0.8.2 | ConfluentIntroduces a new Java producer client that unifies sync and async behavior, improves batching with `linger.ms`, and enhances parallel network communication. Adds a delete topic feature and moves offset management from ZooKeeper to a compacted Kafka topic (__consumer_offsets) for improved scalability. Operational improvements include automated leader rebalancing, controlled shutdown, stronger durability guarantees with `unclean.leader.election` and `min.isr` settings, and connection quotas (`max.connections.per.ip`).