How to Choose the Number of Topics/Partitions in a Kafka Cluster? | Confluent

This post provides guidance on determining the optimal number of partitions in a Kafka cluster. It explains that more partitions generally lead to higher throughput due to parallelism in producers and consumers. It offers a formula based on target throughput and measured per-partition production and consumption rates. The post also discusses the trade-offs of having too many partitions, including increased open file handles, potential unavailability during unclean broker failures (due to controller serial operations and leader election times), increased end-to-end latency (due to replication overhead), and higher memory requirements in clients for buffering. It recommends over-partitioning for future growth and suggests limiting partitions per broker based on cluster size and replication factor to manage latency.