Husky Query Engine

10/1/2025

This post details the architecture and optimizations of Datadog's Husky query engine, enabling real-time access to 100 trillion events. It introduces the four main services: query planner, query orchestrator, metadata service, and reader service. The query planner optimizes, splits queries into time-based steps, and merges results. The query orchestrator fetches fragment metadata, dispatches fragments to reader services (leveraging zone-map pruning for up to 60% reduction in work), and aggregates results. The metadata service abstracts FoundationDB, and reader services fetch data from cache or blob storage. The post highlights the engine's ability to handle schema variations, varying data shapes, and queries spanning millions of fragments.

1/29/2025

This post details the design and implementation of Husky's underlying data storage layer, focusing on efficient compaction strategies. It explains the 'compaction Goldilocks problem' of balancing fragment size for query efficiency and parallelism. The post describes the lazy, multi-criteria approach to triggering compactions and the technical details of the custom columnar storage format designed for efficient streaming of observability data, including support for a large number of columns and embedded skip lists for column offset lookup.

6/28/2024

This post details the evolution of Datadog's timeseries indexing strategy, moving from an original approach of dynamically generating indexes based on query logs to a new strategy focused on pre-computed indices built during ingestion. The original system relied on analyzing query logs to identify slow queries and create indexes for highly selective ones, while also removing obsolete indexes. The new approach involves building indices during the data ingestion process, which significantly improves query performance by reducing the need for runtime index generation and enabling more efficient data retrieval. The post also provides an overview of the metrics platform architecture, including intake, storage (Timeseries Index and Timeseries Database), and query components, and discusses the challenges of timeseries indexing at scale.

4/17/2023

This post details the development of a new, general-purpose query and render scheduler for Datadog's frontend applications, initially to optimize Dashboard performance. The new scheduler simplifies logic by separating query and render scheduling, discarding unnecessary heuristics from the legacy system. The query scheduler uses a fixed time window (2000ms), a task limit (10), and prioritizes visible widgets, while offscreen tasks are ranked by enqueue time. The render scheduler leverages the Browser Scheduling API for efficient task scheduling.

2/22/2023

This post details the evolution of Datadog's Husky event store to achieve exactly-once ingestion and multi-tenancy at scale. It introduces the Shard Router service for deterministic event routing to shards based on event ID and timestamp, leveraging time-bounded Shard Placements to handle dynamic tenant shard assignments. A Sharding Allocator service is used to establish consensus among Shard Router nodes. The system aims to minimize duplicate events and optimize storage costs by limiting the number of tenants processed by individual storage nodes.

5/17/2022

This post introduces Husky, Datadog's third-generation event store, detailing its evolution from previous iterations. It highlights the architectural shift from a coupled storage and clustering system to a decoupled compute and storage architecture. Key technical details include the challenges of storing and querying large, high-cardinality log data compared to metrics, the limitations of the first and second-generation systems (multi-tenant cluster disruption, scaling issues), and the design principles for Husky. These include separating compute from storage for independent scaling, owning the storage engine for flexibility, and enabling features like long-term retention, arbitrary field querying, and advanced aggregation functions. The post also touches upon the inspiration from systems like Snowflake and the need for efficient compaction strategies for managing object store fragments.