Testing & Maintaining Apache Kafka® DR and HA Readiness

This post details the critical aspects of thoroughly testing your Kafka cluster's resilience and implementing effective strategies for long-term DR readiness. It explains the trade-offs of unclean leader election, emphasizing its risk of data loss and recommending it be kept false. It outlines how to handle network partitions using replication factor, ISRs, min.insync.replicas, and acks=all, along with network design and monitoring. The importance of time synchronization via NTP for data consistency, coordinated operations, and DR effectiveness is highlighted. Key metrics for monitoring leader election (LeaderElectionRateAndTimeMs, UncleanLeaderElectionsPerSec, OfflinePartitionsCount, UnderReplicatedPartitions, ActiveControllerCount) and DR health (cross-cluster replication lag, timestamp skew) are provided, along with management strategies like preferred replica election and addressing root causes.