Evolving our real-time timeseries storage again: Built in Rust for performance at scale | Datadog

Datadog built a new real-time timeseries storage engine in Rust, representing the 6th generation of their timeseries storage system. This new engine focuses on high throughput and low latency to handle increasing data volume, complexity, and cardinality. Key achievements include a 60x increase in ingestion performance and 5x faster queries at peak scale. The architecture separates real-time data storage (RTDB) from its index database, with RTDB nodes containing intake, storage engine, snapshot, query execution, and throttling subsystems. The design decisions were driven by limitations encountered in previous generations (Cassandra, Redis, MDBM, Go-based B+ tree, and a previous iteration with DDSketch and RocksDB).