TechPrufer
Sign inGet started

Blogs›Datadog›Live Process and Container Metrics Real-time Data Pipeline

Live Process and Container Metrics Real-time Data Pipeline

Live Process and Container Metrics Real-time Data Pipeline

1
posts
2025

Datadog's live process and container metrics pipeline has been significantly optimized. Initially, it collected data from all hosts in a tenant's infrastructure whenever a user viewed the Processes or Containers pages, leading to millions of data points per second. The system has evolved to collect real-time data only from hosts actively being viewed by the user (up to 50 hosts), drastically reducing data volume. This was achieved by filtering data based on 'host subscriptions' and propagating this state to the intake service via Kafka. Sorting logic was also updated to use standard 10-second interval data instead of high-frequency 2-second data, simplifying the system and improving efficiency. These changes resulted in a 100x reduction in real-time traffic volume, a 98% decrease in infrastructure costs, and lower Datadog Agent resource utilization.

2025

Scaling down to speed up: How we improved efficiency of live process metrics by 100x | Datadog

8/12/2025

This post details the optimization of Datadog's live process and container metrics pipeline. The core technical changes include: 1. Shifting from collecting real-time metrics from all hosts in a tenant to only collecting from actively viewed hosts (up to 50 per user). 2. Implementing a 'host subscription' mechanism on live data servers to filter incoming Kafka payloads based on actively viewed hosts. 3. Propagating host subscription state to the intake service via Kafka to enable early filtering. 4. Updating the sorting logic to use standard 10-second interval data instead of 2-second interval data for live views, simplifying the system and reducing the need for high-frequency data collection. These changes led to significant reductions in data volume, infrastructure costs, and agent resource utilization.