Inside Husky’s query engine: Real-time access to 100 trillion events | Datadog

What this post added

This post details the architecture and optimizations of Datadog's Husky query engine, enabling real-time access to 100 trillion events. It introduces the four main services: query planner, query orchestrator, metadata service, and reader service. The query planner optimizes, splits queries into time-based steps, and merges results. The query orchestrator fetches fragment metadata, dispatches fragments to reader services (leveraging zone-map pruning for up to 60% reduction in work), and aggregates results. The metadata service abstracts FoundationDB, and reader services fetch data from cache or blob storage. The post highlights the engine's ability to handle schema variations, varying data shapes, and queries spanning millions of fragments.