Design a durable key-value store
Design a Durable Key-Value Store
Context
Design a single-node, embeddable key-value store library. This library must provide a simple API and ensure correctness and durability even across process crashes and power failures. Assume the underlying storage is a POSIX-like filesystem on SSD/NVMe with 4 KiB sectors.
API
The key-value store exposes the following API:
put(key, value)get(key)delete(key)
Keys and values are byte arrays. The design should not include secondary indexes or transactions beyond single-key operations.
Requirements
The design must address the following aspects:
- Durability Strategy: Detail the implementation of a write-ahead log (WAL), including its format, rotation policy, and
fsyncstrategy. - On-Disk Data Structures: Describe the organization and layout of data files and metadata on disk.
- Crash Recovery: Explain how to detect and recover from incomplete operations after a crash.
- Compaction and Garbage Collection: Outline mechanisms for reclaiming stale versions of data and tombstones.
fsyncand Batching: Discuss strategies forfsyncand batching, considering throughput and latency trade-offs.- Data Integrity: Address how to handle torn writes and partial files to maintain data integrity.
- Concurrency Model: Define the concurrency model (e.g., single-writer/multi-reader vs. multi-writer; lock-based vs. optimistic/lock-free) and ensure its correctness.
- Consistency Guarantees: Specify the consistency guarantees offered, such as read-after-write, and explore options for linearizability.
- Performance Analysis: Provide an analysis of latency/throughput trade-offs and back-of-the-envelope resource estimates.
- Testing Plan: Propose a minimal testing plan, including failure injection and recovery validation.
AI interview prep
Core dilemma
Balancing high write throughput and low read latency with strict durability, crash recovery, and data integrity guarantees on a single node, given filesystem and hardware constraints.
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Design a durable key-value store
Design a Durable Key-Value Store
Context
Design a single-node, embeddable key-value store library. This library must provide a simple API and ensure correctness and durability even across process crashes and power failures. Assume the underlying storage is a POSIX-like filesystem on SSD/NVMe with 4 KiB sectors.
API
The key-value store exposes the following API:
put(key, value)get(key)delete(key)
Keys and values are byte arrays. The design should not include secondary indexes or transactions beyond single-key operations.
Requirements
The design must address the following aspects:
- Durability Strategy: Detail the implementation of a write-ahead log (WAL), including its format, rotation policy, and
fsyncstrategy. - On-Disk Data Structures: Describe the organization and layout of data files and metadata on disk.
- Crash Recovery: Explain how to detect and recover from incomplete operations after a crash.
- Compaction and Garbage Collection: Outline mechanisms for reclaiming stale versions of data and tombstones.
fsyncand Batching: Discuss strategies forfsyncand batching, considering throughput and latency trade-offs.- Data Integrity: Address how to handle torn writes and partial files to maintain data integrity.
- Concurrency Model: Define the concurrency model (e.g., single-writer/multi-reader vs. multi-writer; lock-based vs. optimistic/lock-free) and ensure its correctness.
- Consistency Guarantees: Specify the consistency guarantees offered, such as read-after-write, and explore options for linearizability.
- Performance Analysis: Provide an analysis of latency/throughput trade-offs and back-of-the-envelope resource estimates.
- Testing Plan: Propose a minimal testing plan, including failure injection and recovery validation.