Design a distributed key-value store at scale
Design a Globally Distributed, Read-Optimized Key-Value Store
Context and Assumptions
Design a globally distributed key-value (KV) store optimized for read-heavy workloads. Assume the following:
- Workload: 90–95% reads, 5–10% writes. Point lookups are dominant, while range scans are rare.
- Data Model: Opaque values associated with a key. Keys can be up to ~1 KB, median value size is ~1 KB, with a maximum value size of 1 MB.
- Scale: The system must handle tens of TB of data per region, support multiple tenants, and operate across multiple regions spanning different continents.
- Latency Targets:
- In-region p99 read latency: ≤ 5–10 ms.
- Global p99 read latency (via nearest region): ≤ 150 ms.
- Availability/Durability:
- Availability: ≥ 99.99%.
- Recovery Point Objective (RPO): 0 within a region.
- Cross-region replication: Tolerated to be asynchronous by default.
Task
Propose an end-to-end design for this key-value store and justify your architectural choices. Your design should explicitly address performance, complexity, and resource trade-offs across the following areas:
- OS-level Performance Considerations: Discuss aspects like threads vs. async I/O, context switching overhead, memory management strategies, and filesystem/kernel tuning.
- Storage Layout and Indexing: Detail the on-disk format, compaction strategy, and trade-offs related to write amplification.
- Partitioning and Sharding: Explain your approach to key distribution, shard sizing, and rebalancing strategies.
- Replication and Caching Layers: Describe the write and read paths, coherence mechanisms, Time-To-Live (TTL) policies, and invalidation strategies.
- Consistency Models and CAP/PACELC Trade-offs: Define the client-visible consistency guarantees and any tunable options available.
- Failure Detection, Fault Isolation, Leader Election, Recovery, and Repair: Outline how the system handles failures and ensures resilience.
- Hotspot Mitigation, Backpressure, and Rate Limiting: Address strategies for managing hot keys, ensuring per-tenant fairness, and controlling overload.
- Capacity Planning, SLAs/SLOs, and Observability: Describe how to plan for capacity, define Service Level Agreements/Objectives, and implement monitoring (metrics, tracing, alerting) to validate the design.
Throughout your explanation, be explicit about your assumptions, call out potential pitfalls and edge cases, and use small numeric examples where helpful to illustrate your points.
AI interview prep
Core dilemma
Achieving extremely low global read latency and high availability for a read-heavy, multi-region workload, while managing eventual consistency trade-offs and ensuring strict regional durability (RPO 0).
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Design a distributed key-value store at scale
Design a Globally Distributed, Read-Optimized Key-Value Store
Context and Assumptions
Design a globally distributed key-value (KV) store optimized for read-heavy workloads. Assume the following:
- Workload: 90–95% reads, 5–10% writes. Point lookups are dominant, while range scans are rare.
- Data Model: Opaque values associated with a key. Keys can be up to ~1 KB, median value size is ~1 KB, with a maximum value size of 1 MB.
- Scale: The system must handle tens of TB of data per region, support multiple tenants, and operate across multiple regions spanning different continents.
- Latency Targets:
- In-region p99 read latency: ≤ 5–10 ms.
- Global p99 read latency (via nearest region): ≤ 150 ms.
- Availability/Durability:
- Availability: ≥ 99.99%.
- Recovery Point Objective (RPO): 0 within a region.
- Cross-region replication: Tolerated to be asynchronous by default.
Task
Propose an end-to-end design for this key-value store and justify your architectural choices. Your design should explicitly address performance, complexity, and resource trade-offs across the following areas:
- OS-level Performance Considerations: Discuss aspects like threads vs. async I/O, context switching overhead, memory management strategies, and filesystem/kernel tuning.
- Storage Layout and Indexing: Detail the on-disk format, compaction strategy, and trade-offs related to write amplification.
- Partitioning and Sharding: Explain your approach to key distribution, shard sizing, and rebalancing strategies.
- Replication and Caching Layers: Describe the write and read paths, coherence mechanisms, Time-To-Live (TTL) policies, and invalidation strategies.
- Consistency Models and CAP/PACELC Trade-offs: Define the client-visible consistency guarantees and any tunable options available.
- Failure Detection, Fault Isolation, Leader Election, Recovery, and Repair: Outline how the system handles failures and ensures resilience.
- Hotspot Mitigation, Backpressure, and Rate Limiting: Address strategies for managing hot keys, ensuring per-tenant fairness, and controlling overload.
- Capacity Planning, SLAs/SLOs, and Observability: Describe how to plan for capacity, define Service Level Agreements/Objectives, and implement monitoring (metrics, tracing, alerting) to validate the design.
Throughout your explanation, be explicit about your assumptions, call out potential pitfalls and edge cases, and use small numeric examples where helpful to illustrate your points.