When failover isn’t safe: Building high-availability PostgreSQL on Kubernetes | Datadog

What this post added

This post details the re-architecture of Datadog's Kubernetes-based PostgreSQL clusters to ensure safe and automatic failover. The previous architecture, which separated leader and read replica pools and used asynchronous replication for standbys, prioritized availability over durability during network partitions, leading to stale data and unsafe failover scenarios. The new architecture leverages synchronous replication for failover candidates, coordinated by Patroni and ZooKeeper, to guarantee data consistency during failover. This involves reconfiguring Patroni to use synchronous replication for standby nodes that are candidates for promotion, ensuring that a standby is sufficiently up-to-date before it can be promoted. The post also discusses the trade-offs between durability and latency, and the validation of the approach through benchmarking and failure testing.