Husky Query Engine

6 posts · May 2022 – Sep 2025

What this trail covers

Datadog's Husky event store has evolved its query engine and underlying data storage layer to handle trillions of events per day. This includes optimizing compaction strategies for efficient merging of small fragments into larger ones, balancing query performance, parallelism, and storage layout. The system uses a columnar storage format similar to Parquet, with custom optimizations for observability data, including a skip list for column offsets to quickly find specific columns. Compaction is performed lazily and efficiently, aiming for a fixed amount of memory and saturating CPU while minimizing storage API requests. The goal is to reduce the number of fragments fetched from object storage and the number of events scanned per query.

All Datadog trails

Oct 2025Inside Husky’s query engine: Real-time access to 100 trillion events | DatadogThis post details the architecture and optimizations of Husky's query engine, focusing on interactive querying of over 100 trillion events. It introduces the four core services: query planner, query orchestrator, metadata service, and reader service. Key technical contributions include: query planner's role in context resolution, validation, throttling, planning, and result aggregation; query orchestrator's metadata fetching, fragment dispatching with up to 60% pruning via zone maps, and aggregation; metadata service's abstraction of FoundationDB for atomic transactions; and reader service's ability to serve data from cache or blob storage. The post also highlights the system's ability to handle schema variations and support 'needle-in-a-haystack' and 'analytics-style' searches.
Jan 2025Husky: Efficient compaction at Datadog scale | DatadogThis post details the design of Husky's underlying data storage layer, focusing on efficient compaction strategies. It explains the 'Goldilocks problem' of finding the optimal fragment size to balance query efficiency, parallelism, and storage layout. The system uses a streaming merge approach for compaction, delaying it until a few dozen fragments are available to merge. It also introduces a custom columnar storage format for efficient streaming, similar to Parquet but optimized for observability data, with a skip list for column offsets. The compaction process is designed to be efficient individually and scalable by issuing only one GET request per input fragment, streaming data in parallel, saturating CPU, and using a fixed amount of memory.
Jun 2024Timeseries indexing at scale | DatadogThis post details the evolution of Datadog's timeseries indexing strategy, moving from an original approach of dynamically generating indexes based on query logs to a new strategy focused on pre-computed indices built during ingestion. The original system relied on analyzing query logs to identify slow queries and create indexes for highly selective ones, while also removing obsolete indexes. The new approach involves building indices during the data ingestion process, which significantly improves query performance by reducing the need for runtime index generation and enabling more efficient data retrieval. The post also provides an overview of the metrics platform architecture, including intake, storage (Timeseries Index and Timeseries Database), and query components, and discusses the challenges of timeseries indexing at scale.
Apr 2023Making fetch happen: Building a general-purpose query and render scheduler | DatadogThis post describes the development of a new, generalized query and render scheduler for Datadog applications. The new scheduler simplifies the logic of the previous system by separating query and render scheduling into discrete modules. The query scheduler uses a new algorithm that prioritizes visible widgets and limits concurrent tasks within fixed time windows (2000ms, 10 task limit). The render scheduler leverages the Browser Scheduling API to schedule any sufficiently costly task, improving browser performance and reducing main thread load.
Feb 2023Husky: Exactly-once ingestion and multi-tenancy at scale | DatadogThis post details the evolution of Datadog's Husky event store's ingestion pipeline, focusing on achieving exactly-once ingestion and multi-tenancy at scale. It introduces the Shard Router service, which uses deterministic routing based on event ID and timestamp to map events to specific shards. To handle dynamic changes in shard assignments and ensure consensus across distributed Shard Router nodes, the system employs time-bounded Shard Placements and a central Sharding Allocator. This approach minimizes storage costs and improves performance by limiting the number of tenants processed by each worker and reducing the overhead associated with logical namespaces.
Advertisement
Jan 2023Performance improvements in the Datadog Agent metrics pipeline | DatadogThis post details performance improvements in the Datadog Agent's metrics pipeline, specifically optimizing the generation of unique metric keys for aggregation. The original algorithm involved sorting and deduplicating tags before hashing, which was identified as a CPU bottleneck. The new approach replaces sorting with XOR bitwise operations for tag hashing and introduces a custom hash set implementation for efficient deduplication, significantly reducing CPU usage and increasing metric processing throughput.
May 2022Introducing Husky, Datadog’s third-generation event store | DatadogThis post details the evolution of Datadog's event store, introducing 'Husky' as the third-generation system. It describes the limitations of previous iterations, particularly the challenges of multi-tenant clusters and the need for independent compute and storage. Husky's design focuses on owning the storage engine from top to bottom, inspired by systems like Snowflake. Key architectural decisions include separating compute from storage to scale ingestion, storage, and query paths independently, and enabling product teams to query and aggregate on any field without pre-defined indexing. The system aims to support critical data retention for longer periods, handle array and windowing functions, and directly store DDSketches for re-aggregation at query time.