You are tasked with scaling an existing synchronous API that lists all files under a directory recursively.
Part A — Scaling Discussion (No Code Required)
Discuss how you would scale the "list all files under a path" API given the following constraints:
- The directory tree can be extremely large, potentially containing millions or billions of files.
- Individual crawls can be very long-running, taking minutes or more, and may time out.
- The system must handle many concurrent user requests for crawls.
- The underlying storage could be local disks, network storage (e.g., NFS), or an object-store-like abstraction.
Your discussion should cover:
- API shape (synchronous vs. asynchronous)
- Pagination and streaming of results
- Concurrency limits and backpressure mechanisms
- Caching and deduplication strategies
- Failure handling and strategies for partial results
Part B — Async Job Implementation (With Provided Templates)
Based on your design from Part A, implement (or describe at a detailed interface level) an asynchronous crawl job. This job must meet the following requirements:
- A client submits a
rootPathand receives ajobIdin response. - Clients can poll for the job's status and fetch its results.
- The job should be robust to failures, support retries, and avoid repeating work excessively.
Specify key components and APIs, for example:
POST /crawl-jobs→{ jobId }GET /crawl-jobs/{jobId}→ status/progressGET /crawl-jobs/{jobId}/results?cursor=...→ paged results
Clearly state any assumptions made during your design (e.g., maximum result size, data retention window, specific storage type).
AI interview prep
Core dilemma
Balancing the need for highly scalable, long-running, fault-tolerant asynchronous processing of potentially massive file system crawls against the requirement for timely status updates and result retrieval for many concurrent users.
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
You are tasked with scaling an existing synchronous API that lists all files under a directory recursively.
Part A — Scaling Discussion (No Code Required)
Discuss how you would scale the "list all files under a path" API given the following constraints:
- The directory tree can be extremely large, potentially containing millions or billions of files.
- Individual crawls can be very long-running, taking minutes or more, and may time out.
- The system must handle many concurrent user requests for crawls.
- The underlying storage could be local disks, network storage (e.g., NFS), or an object-store-like abstraction.
Your discussion should cover:
- API shape (synchronous vs. asynchronous)
- Pagination and streaming of results
- Concurrency limits and backpressure mechanisms
- Caching and deduplication strategies
- Failure handling and strategies for partial results
Part B — Async Job Implementation (With Provided Templates)
Based on your design from Part A, implement (or describe at a detailed interface level) an asynchronous crawl job. This job must meet the following requirements:
- A client submits a
rootPathand receives ajobIdin response. - Clients can poll for the job's status and fetch its results.
- The job should be robust to failures, support retries, and avoid repeating work excessively.
Specify key components and APIs, for example:
POST /crawl-jobs→{ jobId }GET /crawl-jobs/{jobId}→ status/progressGET /crawl-jobs/{jobId}/results?cursor=...→ paged results
Clearly state any assumptions made during your design (e.g., maximum result size, data retention window, specific storage type).