Problem Description
Design a replay feature for an existing distributed publish-subscribe messaging system. This system supports topics, publishers sending messages, and subscribers consuming them. The replay feature must allow subscribers to re-read historical messages, for instance, after a failure, a new deployment, or when a new subscriber joins.
The design must accommodate:
- Replay initiation from a specific timestamp, message ID, or offset.
- Replay functionality for both individual subscribers and subscriber groups.
- Guaranteed message ordering within a partition during replay.
- Robust storage and retention mechanisms for historical messages.
- Maintaining high throughput for normal message delivery concurrently with supporting replay traffic.
- Comprehensive strategies for failure recovery, handling duplicate messages, and addressing operational trade-offs.
Your discussion should detail the proposed APIs, data model, storage architecture, indexing strategy, scaling approach, and an end-to-end explanation of how the replay mechanism functions.
AI interview prep
Core dilemma
Balancing the low-latency, high-throughput requirements of real-time message delivery with the need for efficient, flexible, and scalable historical message retrieval (replay), while ensuring data durability, ordering guarantees, and operational stability in a distributed environment.
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Problem Description
Design a replay feature for an existing distributed publish-subscribe messaging system. This system supports topics, publishers sending messages, and subscribers consuming them. The replay feature must allow subscribers to re-read historical messages, for instance, after a failure, a new deployment, or when a new subscriber joins.
The design must accommodate:
- Replay initiation from a specific timestamp, message ID, or offset.
- Replay functionality for both individual subscribers and subscriber groups.
- Guaranteed message ordering within a partition during replay.
- Robust storage and retention mechanisms for historical messages.
- Maintaining high throughput for normal message delivery concurrently with supporting replay traffic.
- Comprehensive strategies for failure recovery, handling duplicate messages, and addressing operational trade-offs.
Your discussion should detail the proposed APIs, data model, storage architecture, indexing strategy, scaling approach, and an end-to-end explanation of how the replay mechanism functions.