Implement crawler, dedup, and persistent LRU
System Design: Web Crawler with Content Deduplication and Persistent Caching
Design and implement a system that performs web crawling, identifies duplicate content, and utilizes a persistent, extended Least Recently Used (LRU) cache.
Components:
1. Web Crawler
Implement a web crawler that starts from a given URL. The crawler must:
- Recursively explore links found on pages.
- Strictly adhere to the same hostname as the starting URL, ignoring external links.
- Efficiently manage visited URLs to prevent infinite loops and redundant processing.
2. Content Deduplication
Integrate a mechanism to identify duplicate web page content. This component should:
- Process the content of crawled pages.
- Determine if a page's content is identical to previously processed content.
- Provide a way to store and query content hashes or fingerprints for efficient comparison.
3. Persistent LRU Cache
Design and implement an LRU cache that extends standard functionality. This cache must:
- Support variable-length positional and keyword arguments for its keys, ensuring correct cache key generation for complex function calls.
- Include persistence capabilities, allowing the cache's state to be serialized to disk and deserialized upon system restart. This ensures that cached data survives process termination.
The overall system should demonstrate how these three components interact to achieve efficient and robust web content processing.
AI interview prep
Core dilemma
Integrating disparate components (crawler, deduper, persistent cache) into a scalable, fault-tolerant system while managing resource constraints (network I/O, storage, memory) and ensuring data consistency and correctness across restarts.
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Implement crawler, dedup, and persistent LRU
System Design: Web Crawler with Content Deduplication and Persistent Caching
Design and implement a system that performs web crawling, identifies duplicate content, and utilizes a persistent, extended Least Recently Used (LRU) cache.
Components:
1. Web Crawler
Implement a web crawler that starts from a given URL. The crawler must:
- Recursively explore links found on pages.
- Strictly adhere to the same hostname as the starting URL, ignoring external links.
- Efficiently manage visited URLs to prevent infinite loops and redundant processing.
2. Content Deduplication
Integrate a mechanism to identify duplicate web page content. This component should:
- Process the content of crawled pages.
- Determine if a page's content is identical to previously processed content.
- Provide a way to store and query content hashes or fingerprints for efficient comparison.
3. Persistent LRU Cache
Design and implement an LRU cache that extends standard functionality. This cache must:
- Support variable-length positional and keyword arguments for its keys, ensuring correct cache key generation for complex function calls.
- Include persistence capabilities, allowing the cache's state to be serialized to disk and deserialized upon system restart. This ensures that cached data survives process termination.
The overall system should demonstrate how these three components interact to achieve efficient and robust web content processing.