Sign in to track progress, unlock prep layers, and browse your pipeline.Sign in
Problem Description
Design and implement a production-conscious web crawler that fetches pages concurrently using a thread executor. The crawler should prioritize correctness, robustness, and observability while maintaining reasonable simplicity.
Requirements
Input
- Accept one or more seed URLs.
- An optional flag to restrict crawling to the same origin (scheme, host, and port) as the provided seed URLs.
Crawling Behavior
- Fetch pages concurrently using a thread executor, with a configurable maximum worker count.
- Implement a configurable crawl depth limit from each seed URL.
- Extract links from HTML pages and enqueue newly discovered URLs for processing.
- Robustly normalize and resolve links, handling aspects like relative links, URL fragments, default ports, and casing.
- Prevent revisiting the same normalized URL by implementing a deduplication strategy across both in-queue and already visited URLs.
Compliance and Politeness
- Respect
robots.txtrules:- Apply allow/disallow rules based on the user-agent.
- Cache
robots.txtrules per host. - Honor
Crawl-delaydirectives if present.
- Implement per-host politeness and rate limiting, such as allowing at most one request per host per configurable
Xseconds. - Honor the
Retry-Afterheader for HTTP 429 (Too Many Requests) and 503 (Service Unavailable) responses.
Networking
- Gracefully handle HTTP redirects, updating the URL to the final destination and deduplicating based on the normalized final URL.
- Implement robust handling for HTTP errors and timeouts, ensuring the crawler does not crash and applies appropriate backoff strategies.
- Filter fetched content by type, defaulting to
text/html.
Data Structures and Strategy
- Clearly describe the data structures used for the "frontier" (queue of URLs to visit) and the "visited set" (set of already processed URLs).
- Detail the duplicate detection strategy, specifically addressing how it handles URLs that are enqueued versus those already fetched, and how it accounts for redirects.
Testing and Monitoring
- Outline a comprehensive testing strategy, including unit tests, integration tests, concurrency tests, and fault-injection tests.
- Describe what metrics, logs, and alerts would be essential to monitor and measure during a real-world run of the crawler.
Deliverables
- A brief architecture description and its rationale.
- The core algorithm and key components, presented via pseudo-code or a code sketch.
- A clear description of the chosen data structures and the deduplication logic.
- A detailed testing strategy and monitoring plan.
AI interview prep
Core dilemma
Balancing efficient concurrent execution and high throughput with strict politeness rules, robust error handling, and accurate deduplication to ensure correctness and prevent resource exhaustion in a single-process, multi-threaded environment.
Mid-level signals3
Senior signals3
Red flags3
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Problem Description
Design and implement a production-conscious web crawler that fetches pages concurrently using a thread executor. The crawler should prioritize correctness, robustness, and observability while maintaining reasonable simplicity.
Requirements
Input
- Accept one or more seed URLs.
- An optional flag to restrict crawling to the same origin (scheme, host, and port) as the provided seed URLs.
Crawling Behavior
- Fetch pages concurrently using a thread executor, with a configurable maximum worker count.
- Implement a configurable crawl depth limit from each seed URL.
- Extract links from HTML pages and enqueue newly discovered URLs for processing.
- Robustly normalize and resolve links, handling aspects like relative links, URL fragments, default ports, and casing.
- Prevent revisiting the same normalized URL by implementing a deduplication strategy across both in-queue and already visited URLs.
Compliance and Politeness
- Respect
robots.txtrules:- Apply allow/disallow rules based on the user-agent.
- Cache
robots.txtrules per host. - Honor
Crawl-delaydirectives if present.
- Implement per-host politeness and rate limiting, such as allowing at most one request per host per configurable
Xseconds. - Honor the
Retry-Afterheader for HTTP 429 (Too Many Requests) and 503 (Service Unavailable) responses.
Networking
- Gracefully handle HTTP redirects, updating the URL to the final destination and deduplicating based on the normalized final URL.
- Implement robust handling for HTTP errors and timeouts, ensuring the crawler does not crash and applies appropriate backoff strategies.
- Filter fetched content by type, defaulting to
text/html.
Data Structures and Strategy
- Clearly describe the data structures used for the "frontier" (queue of URLs to visit) and the "visited set" (set of already processed URLs).
- Detail the duplicate detection strategy, specifically addressing how it handles URLs that are enqueued versus those already fetched, and how it accounts for redirects.
Testing and Monitoring
- Outline a comprehensive testing strategy, including unit tests, integration tests, concurrency tests, and fault-injection tests.
- Describe what metrics, logs, and alerts would be essential to monitor and measure during a real-world run of the crawler.
Deliverables
- A brief architecture description and its rationale.
- The core algorithm and key components, presented via pseudo-code or a code sketch.
- A clear description of the chosen data structures and the deduplication logic.
- A detailed testing strategy and monitoring plan.