Design real-time per-status device counts
Scenario
You are tasked with designing a system for a fleet of 1,000,000 devices. Each device reports its status once every minute.
Each message contains:
device_idtimestampstatus(one of 10 possible statuses)
Task
Design a system capable of computing, in real-time, the total number of devices for each of the 10 possible statuses. This computation should be performed for each completed minute window (i.e., for every minute, the system should output 10 counts, one for each status).
Clarifications to Address
Your design must explicitly state assumptions and provide solutions for the following data integrity challenges:
- Handling late or out-of-order messages.
- Managing duplicate messages (e.g., due to retries).
- Detecting and accounting for missing heartbeats (i.e., when a device fails to send a message).
- Clearly defining how "previous minute" is determined (e.g., based on event time embedded in the message vs. processing time when the message is received).
Output / Consumption
The computed counts for each minute need to be accessible. Assume these counts will be consumed by a dashboard or an API, requiring access to the current minute's counts and optionally historical data (e.g., for the last 1-24 hours).
AI interview prep
Core dilemma
Balancing real-time, high-volume data aggregation with strict data integrity requirements (deduplication, ordering, late arrivals, missing data) within fixed time windows.
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Design real-time per-status device counts
Scenario
You are tasked with designing a system for a fleet of 1,000,000 devices. Each device reports its status once every minute.
Each message contains:
device_idtimestampstatus(one of 10 possible statuses)
Task
Design a system capable of computing, in real-time, the total number of devices for each of the 10 possible statuses. This computation should be performed for each completed minute window (i.e., for every minute, the system should output 10 counts, one for each status).
Clarifications to Address
Your design must explicitly state assumptions and provide solutions for the following data integrity challenges:
- Handling late or out-of-order messages.
- Managing duplicate messages (e.g., due to retries).
- Detecting and accounting for missing heartbeats (i.e., when a device fails to send a message).
- Clearly defining how "previous minute" is determined (e.g., based on event time embedded in the message vs. processing time when the message is received).
Output / Consumption
The computed counts for each minute need to be accessible. Assume these counts will be consumed by a dashboard or an API, requiring access to the current minute's counts and optionally historical data (e.g., for the last 1-24 hours).