Design GPU inference request batching
Problem: Design GPU Inference Request Batching System
Design a system capable of serving online model-inference requests on GPUs. Clients submit requests one at a time, but GPU throughput is significantly optimized when compatible requests are grouped into batches.
The system must fulfill the following requirements:
- Accept low-latency inference requests.
- Batch compatible requests together efficiently.
- Route processed work to available GPU workers.
- Support multiple distinct models or different versions of the same model.
- Balance overall system throughput against defined latency Service Level Objectives (SLOs).
- Robustly handle system overload, individual component failures, and provide comprehensive observability.
Your design discussion should encompass:
- The external API for clients.
- The internal queueing model.
- The strategy for batching requests.
- The scheduling policy for GPU workers.
- The lifecycle management of GPU workers.
- Signals and mechanisms for autoscaling.
- A thorough analysis of the main architectural trade-offs involved.
AI interview prep
Core dilemma
Balancing the inherent latency of individual online requests with the throughput benefits of GPU batching, while maintaining strict SLOs and handling system dynamics.
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Design GPU inference request batching
Problem: Design GPU Inference Request Batching System
Design a system capable of serving online model-inference requests on GPUs. Clients submit requests one at a time, but GPU throughput is significantly optimized when compatible requests are grouped into batches.
The system must fulfill the following requirements:
- Accept low-latency inference requests.
- Batch compatible requests together efficiently.
- Route processed work to available GPU workers.
- Support multiple distinct models or different versions of the same model.
- Balance overall system throughput against defined latency Service Level Objectives (SLOs).
- Robustly handle system overload, individual component failures, and provide comprehensive observability.
Your design discussion should encompass:
- The external API for clients.
- The internal queueing model.
- The strategy for batching requests.
- The scheduling policy for GPU workers.
- The lifecycle management of GPU workers.
- Signals and mechanisms for autoscaling.
- A thorough analysis of the main architectural trade-offs involved.