Problem Description
Design a distributed service capable of crawling a large file system, starting from a specified root path. Clients should be able to initiate a crawl job via an API. Background workers will asynchronously traverse directories and files. During directory processing, a worker must be able to split the work into smaller tasks and enqueue additional asynchronous jobs, enabling the service to recursively trigger more of its own jobs.
Discussion Points
The design discussion should cover the following aspects:
- API Design: Define APIs for creating a crawl job and checking its status.
- Data Model: Describe the data structures for crawl jobs, individual crawl tasks, and discovered file metadata.
- Recursive Task Scheduling: Explain how workers manage and schedule child tasks recursively.
- Resilience and Reliability: Detail strategies for handling retries, ensuring idempotency, performing deduplication, and managing partial failures within the distributed system.
- Scalability: Address how the system will scale to accommodate extremely large directory trees.
- Core Components: Discuss the roles and interactions of key components such as the database, message queue, and asynchronous workers.
- Client Communication: Explain how progress updates and final results will be exposed to clients.
AI interview prep
Core dilemma
Managing the state and completion of a dynamically expanding, recursive, and long-running distributed workflow while ensuring fault tolerance, scalability, and efficient resource utilization.
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Problem Description
Design a distributed service capable of crawling a large file system, starting from a specified root path. Clients should be able to initiate a crawl job via an API. Background workers will asynchronously traverse directories and files. During directory processing, a worker must be able to split the work into smaller tasks and enqueue additional asynchronous jobs, enabling the service to recursively trigger more of its own jobs.
Discussion Points
The design discussion should cover the following aspects:
- API Design: Define APIs for creating a crawl job and checking its status.
- Data Model: Describe the data structures for crawl jobs, individual crawl tasks, and discovered file metadata.
- Recursive Task Scheduling: Explain how workers manage and schedule child tasks recursively.
- Resilience and Reliability: Detail strategies for handling retries, ensuring idempotency, performing deduplication, and managing partial failures within the distributed system.
- Scalability: Address how the system will scale to accommodate extremely large directory trees.
- Core Components: Discuss the roles and interactions of key components such as the database, message queue, and asynchronous workers.
- Client Communication: Explain how progress updates and final results will be exposed to clients.