TechPrufer
Get started
Confluent
Confluent logoConfluent · Feature trail

Kafka Partition Scalability

8 posts · Mar 2015 – Jan 2026

What this trail covers

Skai successfully migrated their large-scale Kafka deployment to Confluent Cloud by re-architecting their topic strategy to drastically reduce the number of topics. They introduced "united-topic" strategies (single-pipeline, multiple-pipelines, heavy-load) to consolidate events from multiple tables into fewer topics, significantly improving manageability, reducing broker count, and lowering operational costs. Enhanced monitoring and lag handling mechanisms were implemented to address challenges with mixed event streams. The solution also addressed external Kafka client integration through a caching library for dynamic topic mapping. This migration resulted in a >90% reduction in topics without performance degradation, paving the way for further cloud adoption.

All Confluent trails
  1. How a Tier‑1 Bank Tuned Apache Kafka® for p99 Latency for TradingThis post details how a Tier-1 bank achieved sub-5ms p99 latency for critical trading pipelines with Apache Kafka. It covers meticulous architectural discipline, in-depth monitoring using JMX and infrastructure metrics, and nuanced configuration tuning for brokers, producers, and consumers. Key aspects include a "lowest baseline first" methodology using a single partition for strict order preservation, iterative optimization based on metrics like RequestQueueTimeMs and fetch-latency-max, and the use of OpenMessaging Benchmark for reproducible testing. The post also highlights the importance of JVM tuning (Generational ZGC, JDK 21) and infrastructure upgrades for achieving ultra-low latency at high throughput.
  2. Apache Kafka® Cluster Rebalancing Costs More Than You ThinkThis post details the hidden costs of manual Kafka cluster rebalancing, including resource drain (CPU, network, disk I/O) leading to over-provisioning, operational overhead from planning and execution, downtime risk and service instability, and opportunity cost. It highlights how these costs escalate with scale due to exponential complexity, extended disruption windows, and direct spikes in cloud infrastructure bills. The post contrasts manual rebalancing with Confluent's automated, non-disruptive, continuous monitoring and optimization approach, emphasizing predictive scaling and resource efficiency to reduce TCO.
  3. Effectively Migrating Kafka to the Cloud: How Skai Did ItSkai re-architected their Kafka topic strategy to address scalability and cost issues arising from a large number of tenant-specific topics. They introduced three "united-topic" strategies: single-pipeline for low-traffic tables, multiple-pipelines for higher-traffic tables (with dedicated pipelines per destination table), and heavy-load (with multiple partitions) for tables with very high change volumes. This reduced topic count from ~90K to ~1.8K for 600 tenants and 150 tables. They also implemented a lag handling API to dynamically adjust topic resources and a caching library for external Kafka clients to manage dynamic topic assignments. Enhanced monitoring with Datadog metrics was added to identify lag-causing tables within shared topics.
  4. Kafka Clusters and Cluster ManagementThis post addresses the decision-making process for when to create new Apache Kafka clusters. It outlines the physical limits of a single cluster (200,000 partitions or 4,000 partitions per broker) and discusses other limiting factors like network bandwidth, storage, CPU, and memory. It contrasts common but often invalid reasons for multiple clusters (data privacy, noisy neighbors, chargeback, operations overhead, segregation of analytical/transactional data, SLA compromise) with valid reasons (compliance, multi-geographic regions, disaster recovery, independent LOB growth). The post advocates for a single cluster for real-time event correlation and operational economies of scale, while suggesting multiple clusters for specific compliance or geographic needs. It introduces Confluent Cluster Linking and Schema Linking as solutions for inter-cluster data exchange. For operating large clusters, it proposes a federated governance model where a shared services team manages core components (brokers, ZooKeeper, Schema Registry) and LOB teams manage sub-clusters (Kafka Connect, ksqlDB). Key responsibilities for the shared services team include infrastructure provisioning, security, capacity assurance, self-service operations, monitoring, and data governance, with specific details on server security (authentication, TLS, authorization, quotas, audit logs, user provisioning) and capacity assurance (provisioning extra capacity, leveraging Confluent Cloud's elastic scaling).
  5. How to Remove Apache Kafka Brokers the Easy WayThis post details the enhanced broker removal functionality in Confluent Platform 7.0 and Confluent Cloud, which allows for the safe removal of multiple Kafka brokers. It introduces the Self-Balancing Clusters feature as the mechanism for intelligent partition reassignment and load balancing away from brokers being removed. The operation's persistence is managed via a state machine and a Kafka topic, ensuring resilience to failures. A new control-plane API, Broker Replica Exclusion, is introduced to prevent new replica placements on brokers marked for removal, thereby mitigating race conditions during cluster shrinking.
  7. How Pinterest Runs Kafka at ScaleThis post details Pinterest's operational experience running Kafka at scale, focusing on challenges and solutions related to performance and cost control. Key contributions include: 1. **Performance Improvements:** Transitioning from magnetic disks to SSDs to resolve high I/O wait times during broker recovery and load spikes, leading to consistent performance and higher broker densities. 2. **Dynamic Rebalancing Strategy:** Moving from pre-emptive partition reassignments to a static assignment model ('brokerset's) with natural broker recovery to improve availability during replacements. 3. **Message Format Consistency:** Addressing tech debt from mixed message format versions by upgrading clients and cluster defaults to reduce broker CPU load and improve efficiency. 4. **Cost Control:** Implementing rack-aware data transfer strategies for producers and consumers to reduce AWS data transfer costs, enforcing compression on the producer side to lower EC2 and data transfer costs, and optimizing retention policies and replication factors for dev/test clusters to reduce storage and replication costs.
  8. Get Automatic, Self-Balancing Kafka Clusters in Confluent Platform 6.0Introduces Confluent Platform's Self-Balancing Clusters (SBC) feature, which automates Kafka cluster scaling and load balancing. SBC runs on the cluster controller, monitors partition layout and broker load, and automatically reassigns partitions to address imbalances caused by adding/removing brokers, broker failures, or changing traffic patterns. It balances multiple metrics (replica counts, leader counts, disk usage, network usage) while respecting rack awareness and capacity limits. Rebalancing is throttled to minimize impact on cluster traffic. SBC configuration is dynamic, allowing for real-time adjustments to enable/disable balancing and modify throttle settings.
  9. How Tencent PCG Scales Massive Data Pipelines with Apache KafkaThis post details Tencent PCG's development of a 'Federated Kafka' design to scale their massive data pipelines. They built a proxy layer (pproxy and cproxy) that federates multiple Kafka clusters, presenting logical topics to clients and mapping them to physical topics. This enables seamless capacity expansion by adding new clusters without data shuffling, improves fault tolerance with smaller, granular clusters, and allows transparent physical cluster migration. Key components include a name service for proxy discovery and a custom 'controller' service for managing federated metadata. The post outlines workflows for metadata retrieval, cluster provisioning, and decommissioning, and shares operational benchmarks showing rapid cluster initialization and scaling, low metadata refresh latency, and high bandwidth handling. It also notes limitations regarding keyed message ordering and proxy interface evolution, suggesting future integration within Kafka itself.
  10. Scaling Kafka to 10+ GB/Second in Confluent CloudThis post details the internal workings of Confluent Cloud's automated scaling and rebalancing mechanisms for Kafka clusters. It explains the concept of Confluent Units for Kafka (CKUs) as a unit of capacity and how they are used to provision resources. The post elaborates on the partition rebalancing process, including identifying candidate partitions, incrementing ISRs, placing new replicas on new brokers, and changing leaders. It highlights the algorithm used for rebalancing and mentions KIP-455 for replica reassignment. The post emphasizes that this entire process is automated in Confluent Cloud, allowing users to scale their clusters without downtime by simply adjusting the number of CKUs.
  11. Streams and Tables in Apache Kafka: Storage FundamentalsThis post introduces the fundamental storage concepts of Kafka topics and partitions. It explains that topics are unbounded sequences of serialized events, durably stored on brokers, with configurable retention policies. The core concept highlighted is partitioning, where topics are divided into partitions distributed across brokers. This partitioning is crucial for Kafka's scalability, elasticity, and fault tolerance. The post details how producers determine event partitioning using a partitioning function based on event keys to ensure ordering within partitions, and discusses the implications of topic configuration changes and custom partitioning functions on event distribution. It also briefly mentions serialization formats and the role of clients versus brokers in handling them.
  12. Apache Kafka Supports 200K Partitions Per Cluster | ConfluentThis post details significant improvements in Apache Kafka 1.1.0 that increase the number of partitions a cluster can support. Key enhancements include optimizing the controller's controlled shutdown process by using asynchronous ZooKeeper writes and batching leader change communication, reducing shutdown time from 6.5 minutes to 3 seconds. Controller failover time was also improved by using asynchronous ZooKeeper APIs for state reloading, resulting in a 100% improvement in reload time. These changes enable Kafka clusters to support up to 200,000 partitions.
  13. Should You Put Several Event Types in the Same Kafka Topic? | ConfluentThis post introduces the concept that Kafka performance is primarily impacted by the number of partitions, not topics. It advises merging low-throughput topics to reduce partition count and improve latency. It also emphasizes the importance of ordering for events related to the same entity, recommending they be placed in the same topic and partition, especially for event sourcing and microservice communication. The post provides guidelines for when to split or combine topics based on entity relationships, throughput, and consumer needs.
  14. Transactions in Apache Kafka | ConfluentThis post dives deeper into Kafka transactions, building upon previous discussions of exactly-once semantics. It elaborates on the 'why' behind transactions for read-process-write patterns, explains transactional semantics for atomic multi-partition writes and zombie fencing, and details the Java client API for implementing these. It also provides an overview of the internal components like the Transaction Coordinator and Transaction Log, and the data flow involved in transactional operations.
  15. Chain Services with Exactly-Once Guarantees | ConfluentThis post details how Kafka transactions enable exactly-once processing semantics for read-process-write cycles in stream processing applications. It covers atomic multi-partition writes, zombie fencing using transactional.ids, and how consumers read transactional messages. The implementation details include the Transaction Coordinator and Transaction Log, and the data flow for producer-coordinator interaction, transaction initiation, and message production within transactions. It also touches upon the use of idempotence for send-side deduplication.
  16. Exactly-once Semantics is Possible: Here's How Apache Kafka Does itThis post details the introduction of exactly-once semantics (EOS) in Apache Kafka 0.11 and Confluent Platform 3.3. It explains the challenges of achieving EOS in distributed systems and how Kafka addresses them through two key features: idempotent producers (ensuring messages are written only once per partition, even with retries) and transactions (enabling atomic writes across multiple partitions and committing consumer offsets within the same transaction). The post also explains how these features enable exactly-once stream processing via the Kafka Streams API, guaranteeing that for each received record, its processed results are reflected once, even under failures.
  17. Log Compaction – Highlights in the Apache Kafka ® and Stream Processing Community – June 2017 | ConfluentThis post announces the GA of Kafka release 0.11.0.0, highlighting the introduction of Exactly Once Semantics (EOS) which provides exactly once, in-order delivery per partition and atomic writes across multiple partitions. It also notes the increasing adoption of Kafka Connect API and lists several new community-contributed connectors.
  18. Stories from the Front: Lessons Learned from Supporting Apache Kafka ® | ConfluentThis post shares three common operational pitfalls encountered when supporting Apache Kafka deployments: 1) Under-replicated partitions growing inexplicably, often linked to broker performance issues caused by down-level clients or increased load. 2) Full cluster downtime caused by an automated liveness check that blindly restarts brokers without considering the stateful nature of Kafka, leading to extended controller processing times. 3) Terrible performance after adding a new broker, attributed to reassigning a large number of partitions without understanding the data volume involved. High-level recommendations are provided for each scenario, emphasizing root cause analysis for replication issues, human intervention for automation in stateful systems, and careful planning for partition reassignments.
  19. Apache Kafka Hits 1.1 Trillion Messages Per Day - Joins the 4 Comma Club | ConfluentThis post announces LinkedIn's deployment of Apache Kafka reaching 1.1 trillion messages per day, marking a significant milestone in its scalability. It highlights Kafka's role as a horizontally scalable messaging system and its evolution from its creation at LinkedIn in 2010 to powering critical data pipelines, real-time analytics, and stream processing for numerous companies. The post emphasizes Kafka's ability to act as a central nervous system for data, enabling real-time experiences and efficient data synchronization.
  20. Making Apache Kafka Elastic With Apache Mesos | ConfluentThis post details the integration of Apache Kafka with Apache Mesos for elastic cluster management. It explains how to run Kafka brokers as Mesos tasks using a custom Kafka Mesos framework, which includes a scheduler for operational automation and an executor for launching Kafka binaries. The framework provides a REST API and CLI for managing Kafka clusters, including adding, starting, and stopping brokers, and offers better handling of stateful services compared to generic frameworks like Marathon. It also discusses future improvements like auto-scaling clusters and rack-aware partition assignment.
  21. How to Choose the Number of Topics/Partitions in a Kafka Cluster? | ConfluentThis post provides guidance on determining the optimal number of partitions in a Kafka cluster. It explains that more partitions generally lead to higher throughput due to parallelism in producers and consumers. It offers a formula based on target throughput and measured per-partition production and consumption rates. The post also discusses the trade-offs of having too many partitions, including increased open file handles, potential unavailability during unclean broker failures (due to controller serial operations and leader election times), increased end-to-end latency (due to replication overhead), and higher memory requirements in clients for buffering. It recommends over-partitioning for future growth and suggests limiting partitions per broker based on cluster size and replication factor to manage latency.