Testing & Maintaining Apache Kafka® DR and HA Readiness

What this post added

This post provides a deep dive into testing and maintaining Apache Kafka® High Availability (HA) and Disaster Recovery (DR) readiness. It explains the role of unclean leader election, recommending it be kept disabled due to data loss risks. It outlines strategies for handling network partitions using replication factors, ISRs, min.insync.replicas, and acks=all, along with architectural best practices like rack awareness and monitoring. The importance of time synchronization via NTP for data consistency and coordinated operations is highlighted. Key JMX metrics for monitoring leader election health (LeaderElectionRateAndTimeMs, UncleanLeaderElectionsPerSec, OfflinePartitionsCount, UnderReplicatedPartitions, ActiveControllerCount) and DR mechanisms (Cross-Cluster Replication Lag, Timestamp Skew) are detailed. It also emphasizes proactive management through replication, min.insync.replicas, acks=all, keeping unclean.leader.election.enable=false, using preferred replica elections, and addressing root causes of instability.