Design a system that allows users to upload files, asynchronously scan them using multiple engines, and generate a comprehensive report. The system must handle large files (up to multiple GB), support high throughput (tens of millions of files per day), and provide status updates and report retrieval. Security, scalability, and reliability are paramount.
Core Workflow:
- User uploads a file.
- The system initiates one or more scans (e.g., malware, policy/DLP, file-type validation).
- A structured report is generated, including findings, an overall pass/fail status, and metadata.
- Users can query the scan status and download the final report.
Requirements:
- Handle large files (up to multiple GB).
- Scanning is asynchronous; users do not need to maintain an open connection.
- Support multiple scan engines per file, with the possibility of some engines being slow or failing.
- Provide distinct status states:
UPLOADING,QUEUED,SCANNING,COMPLETED,FAILED. - Implement security by default (authentication, authorization, encryption, least privilege).
- Scale to high throughput (tens of millions of files per day).
Deliverables:
- Exposed APIs or UI flows.
- High-level architecture and key components.
- Data model for file metadata, scan jobs, and reports.
- Reliability strategy (retries, idempotency, handling partial failures).
- Observability and operational considerations.
AI interview prep
Core dilemma
Balancing the need for robust, multi-stage asynchronous processing of potentially large files with strict throughput and reliability requirements, while ensuring security and efficient status reporting.
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Design a system that allows users to upload files, asynchronously scan them using multiple engines, and generate a comprehensive report. The system must handle large files (up to multiple GB), support high throughput (tens of millions of files per day), and provide status updates and report retrieval. Security, scalability, and reliability are paramount.
Core Workflow:
- User uploads a file.
- The system initiates one or more scans (e.g., malware, policy/DLP, file-type validation).
- A structured report is generated, including findings, an overall pass/fail status, and metadata.
- Users can query the scan status and download the final report.
Requirements:
- Handle large files (up to multiple GB).
- Scanning is asynchronous; users do not need to maintain an open connection.
- Support multiple scan engines per file, with the possibility of some engines being slow or failing.
- Provide distinct status states:
UPLOADING,QUEUED,SCANNING,COMPLETED,FAILED. - Implement security by default (authentication, authorization, encryption, least privilege).
- Scale to high throughput (tens of millions of files per day).
Deliverables:
- Exposed APIs or UI flows.
- High-level architecture and key components.
- Data model for file metadata, scan jobs, and reports.
- Reliability strategy (retries, idempotency, handling partial failures).
- Observability and operational considerations.