When failover isn’t safe: Building high-availability PostgreSQL on Kubernetes | Datadog

The post details a gameday-induced failure mode where network latency between PostgreSQL primary and replica nodes caused replication lag, making failover unsafe and risking data loss. To address this, Datadog rearchitected its Kubernetes-based PostgreSQL clusters to use synchronous replication for failover candidates, coordinated by Patroni. This change prioritizes data consistency over immediate availability during network latency spikes, ensuring replicas are sufficiently up-to-date before failover. The baseline architecture, Patroni's role with ZooKeeper for coordination, and the trade-offs between availability and durability are discussed.