From Apache Kafka to Confluent Cloud: Optimizing for Speed, Scale, and Storage

What this post added

This post details Confluent's experience optimizing Apache Kafka for Confluent Cloud, covering guiding principles for incremental optimizations and architectural improvements for step-changes in performance. It discusses workload simulation using Trogdor, performance envelopes based on message characteristics, topic-partition distribution, cluster topology, number of connections, and utilization, as well as nightly performance tests. It also covers infrastructure considerations including hardware selection, OS tuning (io_uring, dirty_background_bytes, dirty_bytes), and JVM tuning.