TechPrufer
Sign inGet started

Blogs›Confluent›Kafka Producer and Consumer API Enhancements

Kafka Producer and Consumer API Enhancements

Kafka Producer and Consumer API Enhancements

5
posts
2014–2026

This release details the GA for Kafka release 0.11.0.0, bringing Exactly Once Semantics (EOS) with more stringent idempotent producer semantics, exactly once, in-order delivery per partition, and stronger transactional guarantees with atomic writes across multiple partitions. It also highlights the growing adoption of Kafka Connect API with new community-contributed connectors for AWS S3 sink, Redis source, RabbitMQ source, GitHub source, Amazon Kinesis source, and MongoDB sink. This post details the successful migration of thousands of Kafka clusters from AWS to GCP, involving petabytes of data, using Confluent Replicator for mirroring traffic and Confluent Platform for operational stability. It highlights the shift from batch processing to event streaming, reducing latency from two days to 15 minutes, and improved reliability leading to Kafka not being a suspect for issues. The post also mentions plans for feeding machine learning models with real-time data and using Confluent Cloud for real-time fraud detection, processing over 500,000 daily transactions. It highlights the integration of Confluent Cloud for real-time transaction processing with Apache Spark for batch processing and ML model training. The system analyzes transaction data, customer information, and account details streamed through Confluent Cloud to train ML models and trigger real-time alerts. This has led to a significant reduction in fraud losses and improvements in p99 latency for critical trading pipelines, achieving sub-5ms latency and sustaining 1.6 million messages/sec with messages smaller than 5KB through meticulous tuning of brokers, producers, and consumers, architectural discipline, and in-depth monitoring.

2026

How a Tier‑1 Bank Tuned Apache Kafka® for p99 Latency for Trading

1/23/2026

This post details a Tier-1 bank's successful effort to achieve sub-5ms, p99, end-to-end latency for critical trading pipelines with strict durability requirements. Through architectural discipline, in-depth monitoring, and nuanced configuration of brokers, producers, and consumers, the teams met their goal of 100,000 messages/sec and sustained 1.6 million messages/sec with messages smaller than 5KB. Key technical aspects include using the OpenMessaging Benchmark (OMB) for reproducible testing, focusing on a "lowest baseline first" methodology with a single Kafka partition and four replicas across two data centers, and meticulously instrumenting every stage of the Kafka message path to identify and mitigate tail latency. Specific tuning details are provided for broker metrics (MessagesInPerSec, RequestQueueTimeMs, ISR Health), producer metrics (request-latency-avg, batch-size-avg, linger.ms, batch.size, compression.type, enable.idempotence, acks), and consumer metrics (records-consumed-rate, fetch-latency-avg, max.poll.records, max.poll.interval.ms). Infrastructure upgrades and JVM tuning (Generational ZGC, JDK 21) were also critical.

2020

Top 5 Things Every Apache Kafka Developer Should Know

10/16/2020

This post introduces key improvements to the Kafka producer and consumer APIs. For producers, it explains the `acks` configuration and the importance of `min.insync.replicas` for data durability, and details the new sticky partitioner (KIP-480) for more efficient batching. For consumers, it introduces cooperative rebalancing as an alternative to the older, disruptive rebalance protocol. It also covers the utility of Kafka command-line tools and the use of record headers.

Multi-Threaded Messaging with the Apache Kafka Consumer

8/13/2020

This post introduces a multi-threaded Kafka consumer architecture for Java applications. It details a design that decouples record consumption from processing by using a thread pool to execute tasks for processing records grouped by partition. The post elaborates on the challenges of the single-threaded consumer model, such as `max.poll.interval.ms` timeouts and complex exception handling, and proposes a multi-threaded approach to address these by allowing for longer processing times and simpler retry logic. It emphasizes the importance of maintaining at-least-once delivery semantics and processing order guarantees per partition, and discusses strategies for handling consumer group rebalancing in a multi-threaded context.

2019

Building Systems with Transactions Using Apache Kafka | Confluent

8/20/2019

This post details how to build event-based architectures that retain transactional properties similar to RDBMS by leveraging Apache Kafka, focusing on atomicity, consistency, isolation, and durability. It discusses event modeling, API consistency, and idempotent event processing to achieve transactional guarantees in a distributed system.

2014

What's coming in Apache Kafka 0.8.2 | Confluent

12/1/2014

Introduces a new Java producer client that unifies sync and async behavior, improves batching with `linger.ms`, and enhances parallel network communication. Adds a delete topic feature and moves offset management from ZooKeeper to a compacted Kafka topic (__consumer_offsets) for improved scalability. Operational improvements include automated leader rebalancing, controlled shutdown, stronger durability guarantees with `unclean.leader.election` and `min.isr` settings, and connection quotas (`max.connections.per.ip`).