TechPrufer
Sign inGet started

Blogs›Datadog›Configuration Distribution System

Configuration Distribution System

Configuration Distribution System

2
posts
2016–2025

Datadog developed an internal system to reliably and quickly distribute per-tenant configuration data (context data) to thousands of workload containers. This system addresses challenges of scale, low-latency updates, and resilience, evolving from a Kafka-based invalidation approach to a more robust solution that handles potential failures and high load. The system leverages Consul for configuration distribution and service discovery, with recommendations for optimizing Consul server performance, utilizing ACLs for security, managing watches to prevent self-DDoS, and employing dnsmasq for load lightening.

2025

How we scaled fast, reliable configuration distribution to thousands of workload containers | Datadog

6/17/2025

This post details the evolution of Datadog's system for distributing configuration data to workload containers. It highlights the challenges of scaling configuration updates, the limitations of simpler approaches like on-demand fetching and basic caching with Kafka invalidation, and introduces a new internal system designed for fast, reliable, and resilient configuration distribution at scale. The system aims to abstract operational overhead and adapt to new use cases.

2016

Consul at Datadog | Datadog

8/11/2016

This post details Datadog's use of Consul for distributing configuration and discovering service endpoints in its microservices architecture. It provides recommendations for optimizing Consul server performance (CPU), managing configuration changes via git2consul, securing access with ACLs, preventing self-DDoS with watches, lightening the load with dnsmasq, and the importance of monitoring Consul. It also touches upon the Raft consensus protocol and leadership transitions.