TechPrufer
Sign inGet started

Blogs›Datadog›Real-time Timeseries Storage Engine

Real-time Timeseries Storage Engine

Real-time Timeseries Storage Engine

1
posts
2025

Datadog has evolved its real-time timeseries storage engine for the sixth generation, building a new system in Rust. This iteration focuses on high throughput and low latency, aiming to address challenges posed by increasing data volume, complexity, and cardinality. The new engine is designed to handle high-cardinality workloads, complex queries, and bursty traffic patterns, achieving significant performance gains in ingestion and query speed. The architecture continues to separate real-time data storage (RTDB) from the index database, with RTDB nodes comprising intake, storage engine, snapshot, query execution, and throttling subsystems. This generation represents a significant architectural shift from previous iterations which used Cassandra, Redis, MDBM, Go-based B+ trees, and RocksDB for distribution metrics.

2025

Evolving our real-time timeseries storage again: Built in Rust for performance at scale | Datadog

8/4/2025

This post details the design and implementation of Datadog's 6th generation real-time timeseries storage engine, built from scratch in Rust. It introduces a new architecture focused on high throughput and low latency to handle massive data growth. Key technical decisions include the choice of Rust for performance and safety, a refined separation of RTDB and index databases, and a detailed breakdown of the RTDB node components (intake, storage engine, snapshot, query, throttlers). The post contrasts this new system with its five predecessors, highlighting the engineering trade-offs and performance limitations encountered with Cassandra, Redis, MDBM, a Go-based B+ tree, and RocksDB. The new engine achieves a 60x increase in ingestion performance and 5x faster queries at peak scale.