How a Tier‑1 Bank Tuned Apache Kafka® for p99 Latency for Trading

This post details a Tier-1 bank's successful effort to achieve sub-5ms, p99, end-to-end latency for critical trading pipelines with strict durability requirements. Through architectural discipline, in-depth monitoring, and nuanced configuration of brokers, producers, and consumers, the teams met their goal of 100,000 messages/sec and sustained 1.6 million messages/sec with messages smaller than 5KB. Key technical aspects include using the OpenMessaging Benchmark (OMB) for reproducible testing, focusing on a "lowest baseline first" methodology with a single Kafka partition and four replicas across two data centers, and meticulously instrumenting every stage of the Kafka message path to identify and mitigate tail latency. Specific tuning details are provided for broker metrics (MessagesInPerSec, RequestQueueTimeMs, ISR Health), producer metrics (request-latency-avg, batch-size-avg, linger.ms, batch.size, compression.type, enable.idempotence, acks), and consumer metrics (records-consumed-rate, fetch-latency-avg, max.poll.records, max.poll.interval.ms). Infrastructure upgrades and JVM tuning (Generational ZGC, JDK 21) were also critical.