Design a Task Scheduler for Opaque Long-Running GPU Jobs ("Design Sora")
Design a Task Scheduler for Opaque Long-Running GPU Jobs
Design a distributed system to manage and schedule opaque, long-running, GPU-bound video generation jobs. Users submit text prompts and parameters (resolution, duration, seed), and a black-box binary generates a video, consuming one or more GPUs for a variable duration (seconds to many minutes). This binary cannot be modified or controlled directly.
Your system should handle:
- Accepting user submissions.
- Queuing jobs.
- Scheduling jobs onto a fleet of thousands of GPUs across multiple regions.
- Tracking job lifecycles and handling failures/retries.
- Enforcing fairness and priority across multi-tenant users (e.g., free vs. paid tiers).
- Delivering finished videos back to users.
Constraints & Assumptions:
- 1–5 million jobs per day (average tens/sec, peaks of several hundred/sec).
- Jobs occupy 1+ GPUs, with p50 ~1–2 min, p99 ~10+ min. Jobs are not preemptible.
- Thousands of GPUs available; GPUs are the scarce resource.
- Output videos are tens to hundreds of MB, stored in blob storage and served via signed URLs/CDN.
- Multi-tenant system requiring per-user/per-org quotas and priority enforcement.
- Control plane availability target: 99.9%+.
- Jobs must never be silently lost; failures should lead to retries or explicit failure states.
Similar LeetCode problem: Design Video Sharing Service
AI interview prep
Core dilemma
Balancing high GPU utilization and throughput for expensive, long-running, and unreliable jobs against strict fairness, priority, and availability guarantees for a multi-tenant user base.
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Design a Task Scheduler for Opaque Long-Running GPU Jobs ("Design Sora")
Design a Task Scheduler for Opaque Long-Running GPU Jobs
Design a distributed system to manage and schedule opaque, long-running, GPU-bound video generation jobs. Users submit text prompts and parameters (resolution, duration, seed), and a black-box binary generates a video, consuming one or more GPUs for a variable duration (seconds to many minutes). This binary cannot be modified or controlled directly.
Your system should handle:
- Accepting user submissions.
- Queuing jobs.
- Scheduling jobs onto a fleet of thousands of GPUs across multiple regions.
- Tracking job lifecycles and handling failures/retries.
- Enforcing fairness and priority across multi-tenant users (e.g., free vs. paid tiers).
- Delivering finished videos back to users.
Constraints & Assumptions:
- 1–5 million jobs per day (average tens/sec, peaks of several hundred/sec).
- Jobs occupy 1+ GPUs, with p50 ~1–2 min, p99 ~10+ min. Jobs are not preemptible.
- Thousands of GPUs available; GPUs are the scarce resource.
- Output videos are tens to hundreds of MB, stored in blob storage and served via signed URLs/CDN.
- Multi-tenant system requiring per-user/per-org quotas and priority enforcement.
- Control plane availability target: 99.9%+.
- Jobs must never be silently lost; failures should lead to retries or explicit failure states.
Similar LeetCode problem: Design Video Sharing Service