Design leader election using Redis leases
Design Leader Election using Redis Leases
You are tasked with designing a leader election mechanism for a distributed system. You have N stateless service instances, all running the same job. At any given time, precisely one instance must act as the leader, responsible for performing a singleton task, while all other instances remain as followers or standbys.
You are permitted to use Redis as a coordination dependency. For the purpose of this interview, assume a single primary Redis endpoint is available. You may leverage any Redis features, such as SET NX PX, Lua scripts, or Pub/Sub.
Requirements
- Leader Election: Ensure that at most one instance is designated as the leader at any given moment.
- Failover: If the current leader crashes or becomes unhealthy, a new leader must be elected within a bounded timeframe.
- Lease-based Leadership: Leadership must be time-bounded and require periodic renewal (heartbeating).
- Correctness: Discuss strategies to avoid or mitigate the following issues:
- Split-brain scenarios
- A crashed leader retaining its leadership status ("sticking")
- Assumptions related to clock skew
- Impact of network hiccups or garbage collection pauses on leadership
- Operational Aspects: Address concerns such as:
- Metrics and logging to monitor the system.
- System behavior and recovery mechanisms if Redis becomes unavailable.
Deliverables
- A clear description of the proposed algorithm or protocol.
- The key Redis data model, including specific keys, their Time-To-Live (TTLs), and values.
- Analysis of edge cases and how the system behaves under these conditions.
AI interview prep
Core dilemma
Ensuring strict single-leader correctness (at most one leader) in an asynchronous, distributed environment with unreliable components (network, Redis, service instances), while maintaining high availability (fast failover). The core is balancing correctness (no split-brain) with availability (fast re-election) under various failure modes.
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Design leader election using Redis leases
Design Leader Election using Redis Leases
You are tasked with designing a leader election mechanism for a distributed system. You have N stateless service instances, all running the same job. At any given time, precisely one instance must act as the leader, responsible for performing a singleton task, while all other instances remain as followers or standbys.
You are permitted to use Redis as a coordination dependency. For the purpose of this interview, assume a single primary Redis endpoint is available. You may leverage any Redis features, such as SET NX PX, Lua scripts, or Pub/Sub.
Requirements
- Leader Election: Ensure that at most one instance is designated as the leader at any given moment.
- Failover: If the current leader crashes or becomes unhealthy, a new leader must be elected within a bounded timeframe.
- Lease-based Leadership: Leadership must be time-bounded and require periodic renewal (heartbeating).
- Correctness: Discuss strategies to avoid or mitigate the following issues:
- Split-brain scenarios
- A crashed leader retaining its leadership status ("sticking")
- Assumptions related to clock skew
- Impact of network hiccups or garbage collection pauses on leadership
- Operational Aspects: Address concerns such as:
- Metrics and logging to monitor the system.
- System behavior and recovery mechanisms if Redis becomes unavailable.
Deliverables
- A clear description of the proposed algorithm or protocol.
- The key Redis data model, including specific keys, their Time-To-Live (TTLs), and values.
- Analysis of edge cases and how the system behaves under these conditions.