Husky: Efficient compaction at Datadog scale | Datadog

This post details the design of Husky's underlying data storage layer, focusing on efficient compaction strategies. It explains the 'Goldilocks problem' of finding the optimal fragment size to balance query efficiency, parallelism, and storage layout. The system uses a streaming merge approach for compaction, delaying it until a few dozen fragments are available to merge. It also introduces a custom columnar storage format for efficient streaming, similar to Parquet but optimized for observability data, with a skip list for column offsets. The compaction process is designed to be efficient individually and scalable by issuing only one GET request per input fragment, streaming data in parallel, saturating CPU, and using a fixed amount of memory.