Design a Synchronous Text Classification API
Design a Synchronous Text Classification API
A trained machine-learning model is available to classify text. Design a system that exposes this model through synchronous API endpoints for customers. The design should cover the request flow, load balancing, throttling, failure handling, stateless serving, model deployment, and how the system can scale with increasing traffic and model count.
A database is not strictly required for the inference path. Introduce persistent storage only when a stated requirement necessitates it, and clearly explain the reasoning.
Constraints & Assumptions
- Callers expect a classification response within the request/response cycle.
- Requests can be retried and may arrive in bursts.
- Model artifacts are immutable and versioned.
- Individual serving instances can fail at any time.
- The prompt does not provide specific latency, throughput, payload size, or availability numbers; these should be identified before sizing the system.
AI interview prep
Core dilemma
Balancing low-latency synchronous inference for potentially bursty traffic against the operational complexity of deploying, managing, and scaling multiple ML models and their serving infrastructure.
3 tradeoffs and 2 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Design a Synchronous Text Classification API
Design a Synchronous Text Classification API
A trained machine-learning model is available to classify text. Design a system that exposes this model through synchronous API endpoints for customers. The design should cover the request flow, load balancing, throttling, failure handling, stateless serving, model deployment, and how the system can scale with increasing traffic and model count.
A database is not strictly required for the inference path. Introduce persistent storage only when a stated requirement necessitates it, and clearly explain the reasoning.
Constraints & Assumptions
- Callers expect a classification response within the request/response cycle.
- Requests can be retried and may arrive in bursts.
- Model artifacts are immutable and versioned.
- Individual serving instances can fail at any time.
- The prompt does not provide specific latency, throughput, payload size, or availability numbers; these should be identified before sizing the system.