Blogs›Yahoo›HaloDB Embedded Key-Value Storage Engine
OakMap, a new open-source concurrent ordered key-value map, is introduced. It extends previous research (KiWi) and offers improved scalability and performance over existing solutions like ConcurrentSkipListMap, particularly for big data analytics. OakMap stores keys and values off-heap in memory buffers and uses a two-level on-heap index for efficient access. It addresses garbage collection overhead in Java applications and improves query and update speeds through data locality and in-situ computation. OakMap is being integrated into Druid's core Incremental Index component.
9/20/2018
Introduces HaloDB, a new embedded key-value storage engine built to overcome write performance bottlenecks and long repair times of existing solutions. Details its architecture with append-only log files and an in-memory index in native memory. Presents performance improvements, including a 50% increase in write capacity and submillisecond 99th percentile read latency.
9/13/2018
Introduces OakMap, a new open-source concurrent ordered key-value map designed for big data analytics. Highlights its off-heap memory allocation, two-level on-heap index, and advantages in memory scaling, query speed (scans and lookups), and update speed (data ingestion rate) compared to ConcurrentSkipListMap. Mentions its use in rebuilding Druid's Incremental Index.