Design a Streaming Job Scheduler
Problem Statement
Design a distributed task scheduling service responsible for continuously ingesting tasks and dispatching them to a pool of workers.
Task Definition
Each task is defined by:
id: A unique identifier for the task.deadline: The execution deadline for the task.prerequisites: A list of task IDs that must be successfully completed before this task can be dispatched.
System Requirements
The system must fulfill the following functional and non-functional requirements:
- Task Ingestion: Support ingesting tasks both in batches and as a continuous stream.
- Input Validation: Perform comprehensive validation of incoming task definitions, including:
- Detecting and rejecting malformed input.
- Identifying and handling duplicate task IDs.
- Verifying the validity of all prerequisite references.
- Dependency Management: Ensure that the task dependencies form a Directed Acyclic Graph (DAG) to prevent circular dependencies.
- Scheduling Logic:
- Only dispatch tasks that are "runnable," meaning all their prerequisites have been successfully completed.
- Among all runnable tasks, prioritize dispatching the task with the earliest deadline.
- Worker Interaction:
- Support multiple workers consuming tasks concurrently.
- Guarantee that no single task is assigned to more than one worker simultaneously.
- Fault Tolerance: Implement mechanisms to handle worker crashes, manage task retries, and ensure system recovery.
- Scalability & Observability:
- Scale effectively to accommodate large volumes of tasks.
- Provide robust monitoring capabilities for system health and task status.
Discussion Areas
Candidates should be prepared to discuss the API design, storage model, scheduling logic, DAG validation strategy, concurrency control, fault tolerance mechanisms, and the overall scaling approach.
AI interview prep
Core dilemma
Balancing complex dependency management and strict deadline-based prioritization with high throughput, fault tolerance, and concurrent worker coordination in a distributed environment.
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Design a Streaming Job Scheduler
Problem Statement
Design a distributed task scheduling service responsible for continuously ingesting tasks and dispatching them to a pool of workers.
Task Definition
Each task is defined by:
id: A unique identifier for the task.deadline: The execution deadline for the task.prerequisites: A list of task IDs that must be successfully completed before this task can be dispatched.
System Requirements
The system must fulfill the following functional and non-functional requirements:
- Task Ingestion: Support ingesting tasks both in batches and as a continuous stream.
- Input Validation: Perform comprehensive validation of incoming task definitions, including:
- Detecting and rejecting malformed input.
- Identifying and handling duplicate task IDs.
- Verifying the validity of all prerequisite references.
- Dependency Management: Ensure that the task dependencies form a Directed Acyclic Graph (DAG) to prevent circular dependencies.
- Scheduling Logic:
- Only dispatch tasks that are "runnable," meaning all their prerequisites have been successfully completed.
- Among all runnable tasks, prioritize dispatching the task with the earliest deadline.
- Worker Interaction:
- Support multiple workers consuming tasks concurrently.
- Guarantee that no single task is assigned to more than one worker simultaneously.
- Fault Tolerance: Implement mechanisms to handle worker crashes, manage task retries, and ensure system recovery.
- Scalability & Observability:
- Scale effectively to accommodate large volumes of tasks.
- Provide robust monitoring capabilities for system health and task status.
Discussion Areas
Candidates should be prepared to discuss the API design, storage model, scheduling logic, DAG validation strategy, concurrency control, fault tolerance mechanisms, and the overall scaling approach.