Inside Husky’s query engine: Real-time access to 100 trillion events | Datadog

This post details the architecture and optimizations of Husky's query engine, focusing on interactive querying of over 100 trillion events. It introduces the four core services: query planner, query orchestrator, metadata service, and reader service. Key technical contributions include: query planner's role in context resolution, validation, throttling, planning, and result aggregation; query orchestrator's metadata fetching, fragment dispatching with up to 60% pruning via zone maps, and aggregation; metadata service's abstraction of FoundationDB for atomic transactions; and reader service's ability to serve data from cache or blob storage. The post also highlights the system's ability to handle schema variations and support 'needle-in-a-haystack' and 'analytics-style' searches.