Heatmap Visualization for Distributions

2 posts · Sep 2019 – Apr 2024

What this trail covers

Datadog developed a heatmap visualization system to effectively represent high-resolution distribution metrics over time at arbitrary scale. This system leverages DDSketch for aggregating data and a frontend strategy of sending bins of counts to achieve constant-time rendering. The visualization allows users to identify distinct 'modes' or patterns within data distributions that might be obscured by traditional percentile graphs, enabling deeper analysis of system behavior and performance.

All Datadog trails

May 2024How we built the Datadog heatmap to visualize distributions over time at arbitrary scale | DatadogThis post details the engineering decisions and trade-offs made in building the Datadog heatmap visualization for distributions. It explains the strategy of using DDSketch for scalable aggregation and a frontend approach of sending bins of counts to achieve constant-time rendering. The article highlights how this visualization helps identify distinct 'modes' or patterns within data distributions that traditional percentile graphs might obscure, enabling deeper analysis of system behavior and performance. It contrasts this with percentile graphs, showing how heatmaps can reveal hidden patterns like pulsating low-frequency modes or strong seasonality obscured by aggregation.
Sep 2019Computing accurate percentiles with DDSketch | DatadogThis post introduces DDSketch, a new quantile sketch designed to accurately compute percentiles on large-scale, noisy monitoring data. It addresses the challenges of exact percentile calculation and distributed aggregation by providing a mergeable sketch with relative-error guarantees. The post contrasts DDSketch with existing methods like GK sketches, highlighting how DDSketch reduces approximation errors and improves the accuracy of percentile plots, particularly for high percentiles.
Advertisement