Evolving our real-time timeseries storage again: Built in Rust for performance at scale | Datadog

What this post added

This post details the design and implementation of Datadog's 6th generation real-time timeseries storage engine, built from scratch in Rust. It introduces a new architecture focused on high throughput and low latency to handle massive data growth. Key technical decisions include the choice of Rust for performance and safety, a refined separation of RTDB and index databases, and a detailed breakdown of the RTDB node components (intake, storage engine, snapshot, query, throttlers). The post contrasts this new system with its five predecessors, highlighting the engineering trade-offs and performance limitations encountered with Cassandra, Redis, MDBM, a Go-based B+ tree, and RocksDB. The new engine achieves a 60x increase in ingestion performance and 5x faster queries at peak scale.