Scaling down to speed up: How we improved efficiency of live process metrics by 100x | Datadog

What this post added

This post details the optimization of Datadog's live process and container metrics pipeline. The core technical changes include: 1. Shifting from collecting real-time metrics from all hosts in a tenant to only collecting from actively viewed hosts (up to 50 per user). 2. Implementing a 'host subscription' mechanism on live data servers to filter incoming Kafka payloads based on actively viewed hosts. 3. Propagating host subscription state to the intake service via Kafka to enable early filtering. 4. Updating the sorting logic to use standard 10-second interval data instead of 2-second interval data for live views, simplifying the system and reducing the need for high-frequency data collection. These changes led to significant reductions in data volume, infrastructure costs, and agent resource utilization.