Timeseries indexing at scale | Datadog

What this post added

This post details the evolution of Datadog's timeseries indexing strategy, moving from an original approach of dynamically generating indexes based on query logs to a new strategy focused on pre-computed indices built during ingestion. The original system relied on analyzing query logs to identify slow queries and create indexes for highly selective ones, while also removing obsolete indexes. The new approach involves building indices during the data ingestion process, which significantly improves query performance by reducing the need for runtime index generation and enabling more efficient data retrieval. The post also provides an overview of the metrics platform architecture, including intake, storage (Timeseries Index and Timeseries Database), and query components, and discusses the challenges of timeseries indexing at scale.