Problem Statement
Design a core component within a streaming data pipeline responsible for processing log events, identifying errors, calculating real-time and moving average error rates per service, and triggering alerts based on defined thresholds.
Input
The system receives a continuous stream of log events from multiple upstream services. Each event is guaranteed to contain at least the following fields:
service_id: Identifier for the originating service.timestamp: The time the log event occurred.log_level: The severity level of the log (e.g., INFO, WARN, ERROR, FATAL).message: The actual log content.
Core Tasks
The designed component must fulfill the following requirements:
- Error Filtering: Identify and process only log events classified as errors (e.g.,
log_levelindicating ERROR or FATAL). - Real-time Error Counting: Maintain an up-to-date count of errors for each individual
service_id. - Moving Average Calculation: Compute a moving average of the error count for each
service_idover a specified sliding time window. - Threshold-based Alerting: Trigger an alarm whenever a service's real-time error count or its moving average error rate exceeds a predefined threshold.
Architectural Considerations
Your design should explicitly address:
- The overall system architecture.
- Strategies for state management.
- The chosen windowing approach for moving average calculations.
- Mechanisms for handling late-arriving events.
- Scalability considerations to accommodate increasing event volume and number of services.
- Fault tolerance strategies to ensure system resilience and data integrity.
AI interview prep
Core dilemma
Balancing real-time accuracy and low-latency alerting with the complexities of stateful stream processing, fault tolerance, and handling out-of-order/late events across a potentially high-volume, distributed system.
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Problem Statement
Design a core component within a streaming data pipeline responsible for processing log events, identifying errors, calculating real-time and moving average error rates per service, and triggering alerts based on defined thresholds.
Input
The system receives a continuous stream of log events from multiple upstream services. Each event is guaranteed to contain at least the following fields:
service_id: Identifier for the originating service.timestamp: The time the log event occurred.log_level: The severity level of the log (e.g., INFO, WARN, ERROR, FATAL).message: The actual log content.
Core Tasks
The designed component must fulfill the following requirements:
- Error Filtering: Identify and process only log events classified as errors (e.g.,
log_levelindicating ERROR or FATAL). - Real-time Error Counting: Maintain an up-to-date count of errors for each individual
service_id. - Moving Average Calculation: Compute a moving average of the error count for each
service_idover a specified sliding time window. - Threshold-based Alerting: Trigger an alarm whenever a service's real-time error count or its moving average error rate exceeds a predefined threshold.
Architectural Considerations
Your design should explicitly address:
- The overall system architecture.
- Strategies for state management.
- The chosen windowing approach for moving average calculations.
- Mechanisms for handling late-arriving events.
- Scalability considerations to accommodate increasing event volume and number of services.
- Fault tolerance strategies to ensure system resilience and data integrity.