HaloDB Embedded Key-Value Storage Engine

2 posts · Sep 2018

What this trail covers

OakMap, an open-source concurrent ordered key-value map, has been released. It extends upon earlier research and offers a Java API similar to ConcurrentNavigableMap but scales significantly better with RAM and CPU resources. OakMap stores keys and values in off-heap memory buffers, accessed via a two-level on-heap index. This design alleviates garbage collection overhead, improves data locality for faster lookups and range scans, and optimizes incremental maintenance of complex values like data sketches. Benchmarks show OakMap outperforming ConcurrentSkipListMap in memory utilization, query speed, and update speed. It is being integrated into Apache Druid's core Incremental Index component.

All Yahoo trails

Sep 2018Introducing HaloDB, a fast, embedded key-value storage engine written in JavaThis post introduces HaloDB, detailing its architecture which includes append-only log files and an off-heap in-memory index. It highlights performance improvements, such as a 50% increase in write capacity and sub-millisecond 99th percentile read latency, demonstrated through production data benchmarks. The post also mentions the open-sourcing of HaloDB and encourages community contributions.
Sep 2018Introducing Oak: an Open Source Scalable Key-Value Map for Big Data AnalyticsIntroduces OakMap, a new open-source concurrent ordered key-value map implemented in Java. OakMap utilizes off-heap memory allocation and a two-level index to achieve superior scalability and performance compared to existing solutions like ConcurrentSkipListMap, particularly for analytics workloads. It details the memory scaling benefits due to reduced GC overhead, query speed improvements from data locality, and update speed advantages through in-situ computation on embedded objects. The post also highlights its integration with Apache Druid.
Advertisement