Kafka Partition Scalability

11/8/2018

Introduced significant improvements to the Kafka controller for handling a larger number of partitions per cluster. This involved optimizing controlled broker shutdowns by using asynchronous ZooKeeper writes and batched leader communication, reducing shutdown time from 6.5 minutes to 3 seconds in tests. Controller failover time was also improved by switching to asynchronous ZooKeeper APIs for state reloading, resulting in a 100% improvement in observed reloading time. The post recommends a limit of 4,000 partitions per broker and 200,000 partitions per cluster.

1/18/2018

This post provides guidance on designing Kafka topic structures by discussing the trade-offs between putting multiple event types in the same topic versus splitting them across different topics. It highlights the critical role of ordering within partitions for entities and the performance impact of a large number of topics/partitions. It also touches on schema management considerations for Avro with Confluent Schema Registry.

11/17/2017

This post dives deeper into Kafka transactions, building upon previous discussions of exactly-once semantics. It elaborates on the 'why' behind transactions for read-process-write patterns, explains transactional semantics for atomic multi-partition writes and zombie fencing, and details the Java client API for implementing these. It also provides an overview of the internal components like the Transaction Coordinator and Transaction Log, and the data flow involved in transactional operations.

7/26/2017

This post details how Kafka transactions enable exactly-once processing semantics for read-process-write cycles in stream processing applications. It covers atomic multi-partition writes, zombie fencing using transactional.ids, and how consumers read transactional messages. The implementation details include the Transaction Coordinator and Transaction Log, and the data flow for producer-coordinator interaction, transaction initiation, and message production within transactions. It also touches upon the use of idempotence for send-side deduplication.

6/30/2017

This post details the introduction of exactly-once semantics (EOS) in Apache Kafka 0.11 and Confluent Platform 3.3. It explains the challenges of achieving EOS in distributed systems and how Kafka addresses them through two key features: idempotent producers (ensuring messages are written only once per partition, even with retries) and transactions (enabling atomic writes across multiple partitions and committing consumer offsets within the same transaction). The post also explains how these features enable exactly-once stream processing via the Kafka Streams API, guaranteeing that for each received record, its processed results are reflected once, even under failures.

6/21/2017

This post announces the GA of Kafka release 0.11.0.0, highlighting the introduction of Exactly Once Semantics (EOS) which provides exactly once, in-order delivery per partition and atomic writes across multiple partitions. It also notes the increasing adoption of Kafka Connect API and lists several new community-contributed connectors.

9/1/2015

This post details LinkedIn's deployment of Apache Kafka surpassing 1.1 trillion messages per day, showcasing its evolution from a user activity data pipeline to the company's central nervous system for data. It highlights Kafka's role in powering critical data pipelines, real-time analytics, and stream processing for numerous companies.

3/12/2015

This post provides guidance on choosing the number of topics/partitions in a Kafka cluster. It explains that more partitions generally lead to higher throughput by enabling parallelism on both producer and broker sides, and for consumers. It offers a formula for determining the number of partitions based on target throughput (t) and per-partition production (p) and consumption (c) rates: max(t/p, t/c). It also discusses the trade-off of over-partitioning for keyed messages to maintain ordering guarantees during scaling. The post highlights potential negative impacts of too many partitions, including increased unavailability during broker failures (proportional to the number of partitions on the failed broker) and increased end-to-end latency due to replication overhead. Rules of thumb are provided to limit partitions per broker to avoid these issues. Additionally, it touches upon increased memory requirements in clients (producers and consumers) with more partitions and mentions recent improvements in Kafka client-broker compatibility and data serialization formats.