From Apache Kafka to Confluent Cloud: Optimizing for Speed, Scale, and Storage

This post details Confluent's experience optimizing Apache Kafka for Confluent Cloud, covering guiding principles for incremental optimizations and architectural improvements for step-changes in performance. It discusses workload simulation using Trogdor, performance envelopes based on message characteristics, topic-partition distribution, cluster topology, number of connections, and utilization, as well as nightly performance tests. It also covers infrastructure considerations including hardware selection, OS tuning (io_uring, dirty_background_bytes, dirty_bytes), and JVM tuning.