Design streaming new-vs-returning monthly metrics
Design Streaming Monthly New vs. Returning User Metrics
Problem Description
Design a high-volume streaming system to calculate and emit monthly metrics for "new" versus "returning" users. The system must process an event stream of user requests, each containing at least a user_id, a request_id (unique if available), and an event_time (convertible to a specified time zone). Events are generally time-ordered but can arrive up to 7 days late, and duplicates may occur.
The primary goal is to, for each calendar month in the specified time zone, emit at month close the total counts and percentage shares of requests originating from NEW users versus RETURNING users.
Definitions
- A request is considered NEW if its
event_timefalls within the same calendar month as that user's first-ever recorded request. - A request is considered RETURNING if its
event_timefalls within a calendar month after that user's first-ever recorded request month.
Constraints
- Scale: The system must handle up to 1 billion distinct users and an overall request rate of 50,000 requests per second.
- Memory: Limited state can be kept per user, with an overall constraint of 8 GB RAM available per processing task.
- Accuracy: For approximate designs, the error bounds must ensure a maximum of 0.5 percentage-point error in the final monthly shares.
Tasks
- Data Structures: Propose suitable data structures (e.g., compact first-seen month store, Bloom/Cuckoo filters, HyperLogLog (HLL), Count-Min Sketch) for managing user state and aggregations. Quantify their memory footprints.
- Design Approaches: Provide both exact and approximate design solutions. For approximate solutions, explain how the specified error bounds (≤ 0.5 percentage-point error) are met.
- Event Handling: Explain strategies for handling late and out-of-order events. Define the watermarking and finalization process for monthly aggregations.
- Deduplication: Describe a robust strategy for identifying and handling duplicate events.
- Complexity Analysis: Provide time and space complexity analyses for your proposed solutions.
- Fault Tolerance: Detail recovery and checkpointing mechanisms to ensure fault tolerance.
AI interview prep
Core dilemma
Balancing the need for long-term, per-user state (first-seen month) across a massive user base (1B users) with strict memory limits (8GB per task), while ensuring accurate, event-time-based, windowed aggregations in the face of late, out-of-order, and duplicate events, and meeting specific error bounds.
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Design streaming new-vs-returning monthly metrics
Design Streaming Monthly New vs. Returning User Metrics
Problem Description
Design a high-volume streaming system to calculate and emit monthly metrics for "new" versus "returning" users. The system must process an event stream of user requests, each containing at least a user_id, a request_id (unique if available), and an event_time (convertible to a specified time zone). Events are generally time-ordered but can arrive up to 7 days late, and duplicates may occur.
The primary goal is to, for each calendar month in the specified time zone, emit at month close the total counts and percentage shares of requests originating from NEW users versus RETURNING users.
Definitions
- A request is considered NEW if its
event_timefalls within the same calendar month as that user's first-ever recorded request. - A request is considered RETURNING if its
event_timefalls within a calendar month after that user's first-ever recorded request month.
Constraints
- Scale: The system must handle up to 1 billion distinct users and an overall request rate of 50,000 requests per second.
- Memory: Limited state can be kept per user, with an overall constraint of 8 GB RAM available per processing task.
- Accuracy: For approximate designs, the error bounds must ensure a maximum of 0.5 percentage-point error in the final monthly shares.
Tasks
- Data Structures: Propose suitable data structures (e.g., compact first-seen month store, Bloom/Cuckoo filters, HyperLogLog (HLL), Count-Min Sketch) for managing user state and aggregations. Quantify their memory footprints.
- Design Approaches: Provide both exact and approximate design solutions. For approximate solutions, explain how the specified error bounds (≤ 0.5 percentage-point error) are met.
- Event Handling: Explain strategies for handling late and out-of-order events. Define the watermarking and finalization process for monthly aggregations.
- Deduplication: Describe a robust strategy for identifying and handling duplicate events.
- Complexity Analysis: Provide time and space complexity analyses for your proposed solutions.
- Fault Tolerance: Detail recovery and checkpointing mechanisms to ensure fault tolerance.