Scaling down to speed up: How we improved efficiency of live process metrics by 100x | Datadog

This post details the optimization of Datadog's live process and container metrics pipeline. The core change involved shifting from collecting real-time data from all hosts in a tenant's infrastructure to collecting it only from the hosts actively being viewed by the user (up to 50 hosts). This was achieved by introducing 'host subscriptions' and propagating this state to the intake service via Kafka. The system was further optimized by using 10-second interval data for sorting in live mode, simplifying logic and reducing the need for 2-second interval data for this purpose. These changes resulted in a 100x reduction in real-time traffic volume, 98% infrastructure cost savings, and reduced Datadog Agent resource utilization.