TechPrufer
Sign inGet started

Blogs›Confluent›Kafka Consumer and Group Management

Kafka Consumer and Group Management

Kafka Consumer and Group Management

2
posts
2016–2019

This release details the fundamental mechanics of Kafka consumers and consumer groups, including their role in scalability, availability, and consistency. It explains how consumers pull data, the concept of offsets, and how offsets are managed using the `__consumer_offsets` topic. It also covers the high watermark offset for consistency and the rebalancing process within consumer groups to ensure load balancing and fault tolerance. The post introduces Kafka's membership API, which underpins these functions. The new consumer client in 0.9.0 introduces a consolidated API, reduced dependencies (pure Java, no Zookeeper), and better security support, along with a new group coordination protocol built into Kafka itself, simplifying cross-language client development.

2019

Apache Kafka Data Access Semantics: Consumers and Membership | Confluent

5/7/2019

This post introduces the core concepts of Kafka consumers and consumer groups, explaining their data access semantics. It details the pull-based consumption model, the role of offsets for tracking progress, and the use of the `__consumer_offsets` topic for storing and committing offsets. The post also elaborates on the high watermark offset for ensuring data consistency and the mechanics of consumer group rebalancing for scalability and fault tolerance, highlighting the underlying membership API.

2016

Introducing the Kafka Consumer: Getting Started with the New Apache Kafka 0.9 Consumer Client | Confluent

1/21/2016

Introduces the new Apache Kafka 0.9 consumer client, detailing its advantages over the previous Scala clients: a clean consolidated API, reduced dependencies (pure Java, no Zookeeper), and better security support. It explains the new group coordination protocol built into Kafka, which replaces Zookeeper for group management and simplifies cross-language client development. The post covers core concepts like partitions, consumer groups, offsets, rebalancing, and the high watermark, and provides a tutorial on basic usage, configuration, topic subscription, and the poll loop.