How we use formal modeling, lightweight simulations, and chaos testing to design reliable distributed systems | Datadog

What this post added

This post details the application of formal modeling (TLA+) and lightweight simulations to the design of Datadog's new message queuing service, Courier. It describes how these techniques were used to verify system-level properties and failure modes, particularly focusing on multi-tenancy, at-least-once delivery, graceful degradation, and horizontal scalability. The design of Courier, which uses multiple FoundationDB clusters for tenant isolation and a broker layer with health checks, was analyzed to ensure it met these requirements, incorporating lessons learned from a previous global outage.