Design an image detection system
System Design: End-to-End Image Object-Detection Service
Context
Design a production-grade service that ingests user-uploaded images, runs object detection models, and returns detections via APIs. The system must support both synchronous (low-latency) and asynchronous (high-throughput) use cases. Candidates should state any assumptions made, but for concrete numbers, assume a moderate scale: 1–5k RPS peak, average image size ~1 MB, and typical image dimensions between 640–2048 pixels on the long side.
Requirements
Functional Requirements
- Public APIs for submitting images and retrieving detections.
- Synchronous detection for small, latency-sensitive requests.
- Asynchronous detection for large images or bulk loads.
- Support for idempotency, pagination, and authentication/authorization.
- Result formats must include bounding boxes, classes, and confidences; optional masks.
Non-Functional Requirements
- Accuracy targets (e.g., [email protected], mAP@[0.5:0.95]).
- Latency SLOs (p50/p95 for synchronous vs. asynchronous requests).
- Throughput targets (RPS or jobs/sec).
- Availability (e.g., 99.9%+) and durability.
- Cost constraints.
High-Level Architecture
The design should cover:
- Ingestion: Upload endpoints, pre-signed URLs.
- Storage: Object store for images, database for metadata and results.
- Preprocessing Pipeline: Resize, normalize, EXIF data handling, format conversion.
- Model Serving Tier: GPU inference, batching strategies.
- Asynchronous Workers and Queues: Including Dead Letter Queues (DLQs) and backpressure mechanisms.
- APIs: For submission, status checks, and result retrieval.
- Observability: Metrics, logs, and traces.
Data and Version Management
Address:
- Model registry.
- Dataset versioning.
- Schema evolution.
- Reproducible training and rollbacks (for both models and data).
Performance and Operations
Consider:
- Batching strategy, GPU utilization, and concurrency.
- Autoscaling strategy (request- and queue-driven).
- Caching strategies (for results and model artifacts).
Modeling & ML Ops
Discuss:
- Model choices: single-stage vs. two-stage detectors, and their appropriate use cases.
- Training and labeling pipeline (e.g., active learning, QA processes).
- Evaluation metrics and validation gates.
- Online/offline monitoring (e.g., drift detection, quality checks, SLIs/SLOs).
- A/B testing and rollout/guardrail strategies.
Reliability & Compliance
Detail:
- Failure modes, retries, backpressure, timeouts, circuit breakers.
- Privacy, compliance, data retention, and regionality considerations.
- Cost controls (e.g., GPU choice, right-sizing instances, utilizing spot instances).
- Deployment strategy (e.g., blue/green, canary deployments, rollback procedures).
AI interview prep
Core dilemma
Balancing the conflicting demands of low-latency synchronous inference with high-throughput asynchronous batch processing, while efficiently managing expensive GPU resources and ensuring robust ML model lifecycle management in a production environment.
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Design an image detection system
System Design: End-to-End Image Object-Detection Service
Context
Design a production-grade service that ingests user-uploaded images, runs object detection models, and returns detections via APIs. The system must support both synchronous (low-latency) and asynchronous (high-throughput) use cases. Candidates should state any assumptions made, but for concrete numbers, assume a moderate scale: 1–5k RPS peak, average image size ~1 MB, and typical image dimensions between 640–2048 pixels on the long side.
Requirements
Functional Requirements
- Public APIs for submitting images and retrieving detections.
- Synchronous detection for small, latency-sensitive requests.
- Asynchronous detection for large images or bulk loads.
- Support for idempotency, pagination, and authentication/authorization.
- Result formats must include bounding boxes, classes, and confidences; optional masks.
Non-Functional Requirements
- Accuracy targets (e.g., [email protected], mAP@[0.5:0.95]).
- Latency SLOs (p50/p95 for synchronous vs. asynchronous requests).
- Throughput targets (RPS or jobs/sec).
- Availability (e.g., 99.9%+) and durability.
- Cost constraints.
High-Level Architecture
The design should cover:
- Ingestion: Upload endpoints, pre-signed URLs.
- Storage: Object store for images, database for metadata and results.
- Preprocessing Pipeline: Resize, normalize, EXIF data handling, format conversion.
- Model Serving Tier: GPU inference, batching strategies.
- Asynchronous Workers and Queues: Including Dead Letter Queues (DLQs) and backpressure mechanisms.
- APIs: For submission, status checks, and result retrieval.
- Observability: Metrics, logs, and traces.
Data and Version Management
Address:
- Model registry.
- Dataset versioning.
- Schema evolution.
- Reproducible training and rollbacks (for both models and data).
Performance and Operations
Consider:
- Batching strategy, GPU utilization, and concurrency.
- Autoscaling strategy (request- and queue-driven).
- Caching strategies (for results and model artifacts).
Modeling & ML Ops
Discuss:
- Model choices: single-stage vs. two-stage detectors, and their appropriate use cases.
- Training and labeling pipeline (e.g., active learning, QA processes).
- Evaluation metrics and validation gates.
- Online/offline monitoring (e.g., drift detection, quality checks, SLIs/SLOs).
- A/B testing and rollout/guardrail strategies.
Reliability & Compliance
Detail:
- Failure modes, retries, backpressure, timeouts, circuit breakers.
- Privacy, compliance, data retention, and regionality considerations.
- Cost controls (e.g., GPU choice, right-sizing instances, utilizing spot instances).
- Deployment strategy (e.g., blue/green, canary deployments, rollback procedures).