TechPrufer
Sign inGet started

Blogs›Datadog›Multi-tenant Data Replication Platform

Multi-tenant Data Replication Platform

Multi-tenant Data Replication Platform

3
posts
2025–2026

Datadog built a managed data replication platform to deliver highly reliable, highly scalable, and flexible data movement across the company. This platform abstracts operational overhead, provides robust monitoring and alerting, and adapts to new use cases. It evolved from addressing performance issues in a shared Postgres database by rerouting search queries to a dedicated search platform with dynamically denormalized data. The provisioning process was automated using Temporal workflows, breaki This post details the re-architecture of Datadog's Kubernetes-based PostgreSQL clusters to ensure safe and automatic failover. The previous architecture, which separated leader and read replica pools and used asynchronous replication for standbys, prioritized availability over durability during network partitions, leading to stale data and unsafe failover scenarios. The new architecture leverages synchronous replication for failover candidates, coordinated by Patroni and ZooKeeper, to guarantee data consistency during failover. This involves reconfiguring Patroni to use synchronous replication for standby nodes that are candidates for promotion, ensuring that a standby is sufficiently up-to-date before it can be promoted. The post also discusses the trade-offs between durability and latency, and the validation of the approach through benchmarking and failure testing.

2026

When failover isn’t safe: Building high-availability PostgreSQL on Kubernetes | Datadog

6/4/2026

This post details the re-architecture of Datadog's Kubernetes-based PostgreSQL clusters to ensure safe and automatic failover. The previous architecture, which separated leader and read replica pools and used asynchronous replication for standbys, prioritized availability over durability during network partitions, leading to stale data and unsafe failover scenarios. The new architecture leverages synchronous replication for failover candidates, coordinated by Patroni and ZooKeeper, to guarantee data consistency during failover. This involves reconfiguring Patroni to use synchronous replication for standby nodes that are candidates for promotion, ensuring that a standby is sufficiently up-to-date before it can be promoted. The post also discusses the trade-offs between durability and latency, and the validation of the approach through benchmarking and failure testing.

2025

Replication redefined: How we built a low-latency, multi-tenant data replication platform | Datadog

11/4/2025

This post details the creation of a managed, multi-tenant data replication platform at Datadog. It describes the initial problem of scaling a shared Postgres database for search queries, leading to the architectural decision to replicate data to a dedicated search platform, reducing page load times by 97%. It then covers the automation of replication pipeline provisioning using Temporal workflows, breaking down complex steps into modular tasks. Finally, it discusses the choice of asynchronous replication for low-latency, high-reliability data movement, contrasting it with synchronous replication and highlighting its benefits for multi-tenant systems.

Breaking up a monolith: How we’re unwinding a shared database at scale | Datadog

6/17/2025

This post details Datadog's strategy and execution for unwinding a large, shared monolithic database into independently owned instances. It outlines a three-phase plan: logical separation (identifying data domains and responsibilities using DBM query samples and engineer collaboration), access reduction (minimizing direct database access by enforcing API/gRPC service usage and implementing a homegrown PG Proxy for flexible routing), and physical separation (migrating data to new, independently owned instances managed by the OrgStore platform). The post emphasizes the importance of platform investments like Rapid (for service building) and OrgStore (for managed databases) in making such migrations feasible and safe.