Introducing Husky, Datadog’s third-generation event store | Datadog

What this post added

This post introduces Husky, Datadog's third-generation event store, detailing its evolution from previous iterations. It highlights the architectural shift from a coupled storage and clustering system to a decoupled compute and storage architecture. Key technical details include the challenges of storing and querying large, high-cardinality log data compared to metrics, the limitations of the first and second-generation systems (multi-tenant cluster disruption, scaling issues), and the design principles for Husky. These include separating compute from storage for independent scaling, owning the storage engine for flexibility, and enabling features like long-term retention, arbitrary field querying, and advanced aggregation functions. The post also touches upon the inspiration from systems like Snowflake and the need for efficient compaction strategies for managing object store fragments.