TechPrufer
Get started
Spotify
Spotify logoSpotify · Feature trail

Sparkey Key-Value Store

1 post · Aug 2013

What this trail covers

Spotify has open-sourced Sparkey, a C-based key-value store inspired by CDB and Bitcask. It addresses the limitations of CDB's 4GB file size limit and Tokyo Cabinet's overhead and locking issues. Sparkey offers fast lookups, low overhead for small key-value pairs, block-level compression, and fast creation times. It consists of a log file and an on-disk hash table index. The system has been used internally for over a year on the content metadata lookup service and is being adopted by newer services.

All Spotify trails
  1. New open source key-value store: sparkeyThis post introduces Sparkey, a new open-source key-value store developed by Spotify. It details the historical context leading to its creation, comparing it to CDB and Tokyo Cabinet. The post explains Sparkey's architecture, which involves a log file and an on-disk hash table, and discusses its performance characteristics, including insertion and lookup throughput. It also provides links to the GitHub repository for further details on the file format and performance benchmarks.