Design crawler storing only image URLs
Design an Image-URL Crawler
Design a production web crawler that fetches HTML pages and extracts only image URLs. The crawler should not store the full HTML bodies. Image URLs can be found in:
<img src="..." />tags<source srcset="..." />within<picture>elements- Inline CSS styles (e.g.,
style="background-image: url('...')")
Assume this crawler will run continuously at scale and must support query APIs. Key components to consider include:
- URL frontier/scheduler
- Fetchers
- Parsers
- Deduplication mechanisms
- Storage and indexing for image URLs and their associated page metadata
- Control plane for managing the crawl process
Additionally, address crawl politeness and compliance, parsing at scale, deduplication strategies, storage design, query API design, sharding and scaling, fault tolerance, monitoring, and capacity planning. Consider data retention and privacy.
AI interview prep
Core dilemma
Balancing the immense scale of web crawling and URL extraction with the need for efficient, low-latency storage and querying of image URLs, while strictly adhering to politeness policies and fault tolerance.
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Design crawler storing only image URLs
Design an Image-URL Crawler
Design a production web crawler that fetches HTML pages and extracts only image URLs. The crawler should not store the full HTML bodies. Image URLs can be found in:
<img src="..." />tags<source srcset="..." />within<picture>elements- Inline CSS styles (e.g.,
style="background-image: url('...')")
Assume this crawler will run continuously at scale and must support query APIs. Key components to consider include:
- URL frontier/scheduler
- Fetchers
- Parsers
- Deduplication mechanisms
- Storage and indexing for image URLs and their associated page metadata
- Control plane for managing the crawl process
Additionally, address crawl politeness and compliance, parsing at scale, deduplication strategies, storage design, query API design, sharding and scaling, fault tolerance, monitoring, and capacity planning. Consider data retention and privacy.