Problem Description
Design a resilient system to compute payouts for delivery drivers (Dashers) based on a continuous stream of order lifecycle events. The event stream consists of two primary event types:
ACCEPT(orderId, dasherId, timestamp): Indicates when a Dasher accepts an order.FULFILL(orderId, dasherId, timestamp): Indicates when a Dasher successfully fulfills an order.
Assume that for the happy path, input events are valid: each order is accepted and subsequently fulfilled by the same Dasher exactly once, and events arrive in the correct chronological order (ACCEPT then FULFILL). The system must be designed for high write throughput, ensure financial correctness, and proactively handle various failure modes.
Core Requirements
1. Event Ingestion and Semantics
- Define the ingestion semantics for
ACCEPTandFULFILLevents. - Explain the concepts of idempotency, deduplication, and the trade-offs between achieving exactly-once versus at-least-once delivery guarantees.
2. Data Models and Storage
- Propose data models capable of supporting:
- High write throughput for both raw events and derived state.
- Efficient querying of Dasher payouts for specific pay periods.
- Capabilities for backfills and reprocessing of historical data.
- Specify appropriate storage choices (e.g., event log, OLTP/NoSQL databases, data lake), including partitioning and indexing strategies.
3. Computation of Payouts
- Describe the logic for how payouts are computed from the order lifecycle events (e.g., payout triggered at FULFILL, adjustments, etc.).
- Outline the choice between batch and streaming computation methods, how a pay period is finalized, and how backfills and reconciliation processes are supported.
4. Data Quality and Remediation
- Proactively identify and propose strategies to detect and remediate common data quality issues and failures:
- A
FULFILLevent arriving without a correspondingACCEPTevent. - An
ACCEPTevent that never receives a subsequentFULFILLevent. - Duplicate event occurrences.
- Out-of-order event delivery.
- A
- Define Service Level Agreements (SLAs), the use of dead-letter queues, and specific correction strategies for these scenarios.
5. APIs
- Define APIs for:
- Ingesting
ACCEPTandFULFILLevents. - Querying a Dasher's payout for a specified local pay period.
- Ingesting
- Explain how the system will handle complexities related to timezones and Daylight Saving Time (DST) for payout calculations and queries.
6. Monitoring, Alerting, and Correctness
- Define key metrics, alerting mechanisms, and correctness checks essential for the system's operation.
- Be explicit about the trade-offs made and the consistency guarantees provided by the design.
AI interview prep
Core dilemma
Achieving financial accuracy and auditability (effectively 'exactly-once' processing for payouts) from a high-throughput, potentially out-of-order, and at-least-once event stream, while maintaining low-latency query capabilities and robust failure handling.
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Problem Description
Design a resilient system to compute payouts for delivery drivers (Dashers) based on a continuous stream of order lifecycle events. The event stream consists of two primary event types:
ACCEPT(orderId, dasherId, timestamp): Indicates when a Dasher accepts an order.FULFILL(orderId, dasherId, timestamp): Indicates when a Dasher successfully fulfills an order.
Assume that for the happy path, input events are valid: each order is accepted and subsequently fulfilled by the same Dasher exactly once, and events arrive in the correct chronological order (ACCEPT then FULFILL). The system must be designed for high write throughput, ensure financial correctness, and proactively handle various failure modes.
Core Requirements
1. Event Ingestion and Semantics
- Define the ingestion semantics for
ACCEPTandFULFILLevents. - Explain the concepts of idempotency, deduplication, and the trade-offs between achieving exactly-once versus at-least-once delivery guarantees.
2. Data Models and Storage
- Propose data models capable of supporting:
- High write throughput for both raw events and derived state.
- Efficient querying of Dasher payouts for specific pay periods.
- Capabilities for backfills and reprocessing of historical data.
- Specify appropriate storage choices (e.g., event log, OLTP/NoSQL databases, data lake), including partitioning and indexing strategies.
3. Computation of Payouts
- Describe the logic for how payouts are computed from the order lifecycle events (e.g., payout triggered at FULFILL, adjustments, etc.).
- Outline the choice between batch and streaming computation methods, how a pay period is finalized, and how backfills and reconciliation processes are supported.
4. Data Quality and Remediation
- Proactively identify and propose strategies to detect and remediate common data quality issues and failures:
- A
FULFILLevent arriving without a correspondingACCEPTevent. - An
ACCEPTevent that never receives a subsequentFULFILLevent. - Duplicate event occurrences.
- Out-of-order event delivery.
- A
- Define Service Level Agreements (SLAs), the use of dead-letter queues, and specific correction strategies for these scenarios.
5. APIs
- Define APIs for:
- Ingesting
ACCEPTandFULFILLevents. - Querying a Dasher's payout for a specified local pay period.
- Ingesting
- Explain how the system will handle complexities related to timezones and Daylight Saving Time (DST) for payout calculations and queries.
6. Monitoring, Alerting, and Correctness
- Define key metrics, alerting mechanisms, and correctness checks essential for the system's operation.
- Be explicit about the trade-offs made and the consistency guarantees provided by the design.