Make Cache Fetch Thread-Safe and Single-Flight
Make Cache Fetch Thread-Safe and Single-Flight
A Time-To-Live (TTL) cache has a fetch(key, loader, ttl) operation. This operation should immediately return a valid cached value if one exists. Otherwise, it should call the provided loader function, cache a successful result, and then return it. Loader errors must not be cached. The challenge is to make this fetch operation safe and efficient when multiple threads simultaneously request the same key.
Constraints & Assumptions:
- The
loadercan take 100 milliseconds or longer and may fail or time out. - Different keys should generally be loaded concurrently.
- The cache supports standard
put,get, anddeleteoperations in addition tofetch. - A key can be replaced or deleted even while a load for that key is in progress.
- Values, once published to the cache, are immutable.
Problem Breakdown:
- Thread-Safe Core State: Define the synchronization mechanisms for the cache's internal state, including the value map, TTL metadata, and any structures used for recency or cleanup. Explain the linearization point for each cache operation.
- Fetch and Key Conflicts: Design a mechanism to prevent a "miss storm" (many concurrent requests for the same key causing redundant loader calls) while still allowing concurrent loads for different keys. This part must correctly handle loader success, errors, timeouts, and race conditions with
deleteor replacement operations.
AI interview prep
Core dilemma
Balancing fine-grained concurrency for high throughput across many keys against the need to serialize expensive, potentially failing I/O operations for a single key to avoid redundant work and resource exhaustion.
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Make Cache Fetch Thread-Safe and Single-Flight
Make Cache Fetch Thread-Safe and Single-Flight
A Time-To-Live (TTL) cache has a fetch(key, loader, ttl) operation. This operation should immediately return a valid cached value if one exists. Otherwise, it should call the provided loader function, cache a successful result, and then return it. Loader errors must not be cached. The challenge is to make this fetch operation safe and efficient when multiple threads simultaneously request the same key.
Constraints & Assumptions:
- The
loadercan take 100 milliseconds or longer and may fail or time out. - Different keys should generally be loaded concurrently.
- The cache supports standard
put,get, anddeleteoperations in addition tofetch. - A key can be replaced or deleted even while a load for that key is in progress.
- Values, once published to the cache, are immutable.
Problem Breakdown:
- Thread-Safe Core State: Define the synchronization mechanisms for the cache's internal state, including the value map, TTL metadata, and any structures used for recency or cleanup. Explain the linearization point for each cache operation.
- Fetch and Key Conflicts: Design a mechanism to prevent a "miss storm" (many concurrent requests for the same key causing redundant loader calls) while still allowing concurrent loads for different keys. This part must correctly handle loader success, errors, timeouts, and race conditions with
deleteor replacement operations.