Husky: Efficient compaction at Datadog scale | Datadog

What this post added

This post details the design and implementation of Husky's underlying data storage layer, focusing on efficient compaction strategies. It explains the 'compaction Goldilocks problem' of balancing fragment size for query efficiency and parallelism. The post describes the lazy, multi-criteria approach to triggering compactions and the technical details of the custom columnar storage format designed for efficient streaming of observability data, including support for a large number of columns and embedded skip lists for column offset lookup.