Design a News Aggregation System (Google News-style)
Design a News Aggregation System
Design a large-scale news aggregation service similar to Google News. The system should continuously discover and crawl news articles from tens of thousands of publishers across the web. Key functionalities include:
- Crawling: Discover and fetch articles from a vast number of sources (RSS, sitemaps, publisher APIs, raw HTML).
- Story Clustering: Group articles that cover the same real-world event into a single "story".
- Trending Topics: Surface "hot topics" in near real-time.
- News Feed: Serve a ranked, deduplicated news feed to readers.
The focus is on the ingestion and processing pipeline: crawling at scale, coordinating a fleet of crawler machines with fault tolerance, deduplicating and clustering articles, detecting hot topics, and selecting appropriate storage solutions. The design should consider the impact of these choices on freshness, consistency, and latency.
Constraints & Assumptions:
- Sources: Approximately 50,000–100,000 news sources/feeds.
- Ingest Volume: 2–5 million new articles per day, with potential for 5–10x bursts during major events.
- Readers: 50 million daily active users, leading to tens of thousands of feed reads per second at peak.
- Freshness: Breaking news should appear in feeds within ~2–5 minutes of publication.
- Latency: p95 feed read latency should be less than ~200 ms.
- Politeness: Respect
robots.txtand per-domain crawl-rate limits. - Storage: Retain the article corpus for years, serving recent feeds from optimized storage.
- Assume storing article metadata and a snippet/link, not full reproductions.
Clarifying Questions to Ask:
- Scope: Global multi-language/region, or single language/region initially?
- Personalized feeds or a single editorial/regional feed for v1?
- Is full-text search in scope, or only the browse/feed + trending experience?
- Do we host article text or only link out with a snippet?
- How fresh must trending topics be (sub-minute vs. few minutes)?
- Multimedia (images/video) in scope, or text articles only?
AI interview prep
Core dilemma
Balancing the scale and real-time requirements of a global news ingestion and processing pipeline against the need for politeness, fault tolerance, and efficient storage of a massive, ever-growing corpus.
4 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Design a News Aggregation System (Google News-style)
Design a News Aggregation System
Design a large-scale news aggregation service similar to Google News. The system should continuously discover and crawl news articles from tens of thousands of publishers across the web. Key functionalities include:
- Crawling: Discover and fetch articles from a vast number of sources (RSS, sitemaps, publisher APIs, raw HTML).
- Story Clustering: Group articles that cover the same real-world event into a single "story".
- Trending Topics: Surface "hot topics" in near real-time.
- News Feed: Serve a ranked, deduplicated news feed to readers.
The focus is on the ingestion and processing pipeline: crawling at scale, coordinating a fleet of crawler machines with fault tolerance, deduplicating and clustering articles, detecting hot topics, and selecting appropriate storage solutions. The design should consider the impact of these choices on freshness, consistency, and latency.
Constraints & Assumptions:
- Sources: Approximately 50,000–100,000 news sources/feeds.
- Ingest Volume: 2–5 million new articles per day, with potential for 5–10x bursts during major events.
- Readers: 50 million daily active users, leading to tens of thousands of feed reads per second at peak.
- Freshness: Breaking news should appear in feeds within ~2–5 minutes of publication.
- Latency: p95 feed read latency should be less than ~200 ms.
- Politeness: Respect
robots.txtand per-domain crawl-rate limits. - Storage: Retain the article corpus for years, serving recent feeds from optimized storage.
- Assume storing article metadata and a snippet/link, not full reproductions.
Clarifying Questions to Ask:
- Scope: Global multi-language/region, or single language/region initially?
- Personalized feeds or a single editorial/regional feed for v1?
- Is full-text search in scope, or only the browse/feed + trending experience?
- Do we host article text or only link out with a snippet?
- How fresh must trending topics be (sub-minute vs. few minutes)?
- Multimedia (images/video) in scope, or text articles only?