Real-time Timeseries Storage Engine

Datadog has evolved its real-time timeseries storage engine for the sixth generation, building a new system in Rust. This iteration focuses on high throughput and low latency, aiming to address challenges posed by increasing data volume, complexity, and cardinality. The new engine is designed to handle high-cardinality workloads, complex queries, and bursty traffic patterns, achieving significant performance gains in ingestion and query speed. The architecture continues to separate real-time data. This post details the evolution of the timeseries indexing service, comparing two generations of indexing services and their impact on query performance and maintenance. It highlights the challenges of timeseries indexing at scale, including the trade-offs between query performance, reliability, and maintenance overhead. The new indexing strategy focuses on optimizing index selectivity and managing index lifecycle to improve query efficiency and reduce operational burden.