How to Choose the Number of Topics/Partitions in a Kafka Cluster? | Confluent

What this post added

This post provides guidance on choosing the number of topics/partitions in a Kafka cluster. It explains that more partitions generally lead to higher throughput by enabling parallelism on both producer and broker sides, and for consumers. It offers a formula for determining the number of partitions based on target throughput (t) and per-partition production (p) and consumption (c) rates: max(t/p, t/c). It also discusses the trade-off of over-partitioning for keyed messages to maintain ordering guarantees during scaling. The post highlights potential negative impacts of too many partitions, including increased unavailability during broker failures (proportional to the number of partitions on the failed broker) and increased end-to-end latency due to replication overhead. Rules of thumb are provided to limit partitions per broker to avoid these issues. Additionally, it touches upon increased memory requirements in clients (producers and consumers) with more partitions and mentions recent improvements in Kafka client-broker compatibility and data serialization formats.