System Design: News Aggregator
Design a news aggregator system, similar to a "Top Stories" or Google News product, that collects articles from numerous publishers and delivers ranked feeds to users.
Core Requirements
- Ingestion: The system must ingest articles from thousands of diverse sources, including RSS/Atom feeds, publisher APIs, and webhooks.
- Normalization & Storage: Article content and metadata (title, body/snippet, author, publish time, canonical URL, source, topics/tags) need to be normalized and stored efficiently.
- De-duplication: Identify and de-duplicate near-identical stories that may appear across multiple sources, often reporting on the same event.
- Ranking & Serving:
- Provide a global "homepage" feed with universally ranked stories.
- Offer topic-specific feeds (e.g., Sports, Technology).
- Optionally, support personalized feeds tailored to individual user interests.
- Performance: Ensure low-latency reads for feed browsing, with a strong emphasis on freshness, meaning new stories should appear quickly.
Non-Functional Requirements
- Availability: The system must be highly available, supporting multi-region reads.
- Scalability: It should be capable of handling significant traffic spikes, particularly during breaking news events.
- Content Safety: Implement reasonable measures for basic spam detection and handling of malicious sources.
Scope of Discussion
Candidates should cover the following architectural aspects, making reasonable assumptions about traffic, QPS, and data volume as needed:
- API design for both article ingestion and feed consumption.
- Data model and choices for persistent storage.
- The complete ingestion and processing pipeline, including parsing, content enrichment, and de-duplication.
- Approaches to ranking, considering various signals and the balance between batch and real-time processing.
- Strategies for caching and efficient feed generation.
- Considerations for system reliability, handling backfills, and comprehensive monitoring.
AI interview prep
Core dilemma
Balancing the freshness and low-latency serving of highly dynamic, personalized content against the complexity and cost of ingesting, processing, and de-duplicating massive volumes of data from diverse sources in real-time.
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
System Design: News Aggregator
Design a news aggregator system, similar to a "Top Stories" or Google News product, that collects articles from numerous publishers and delivers ranked feeds to users.
Core Requirements
- Ingestion: The system must ingest articles from thousands of diverse sources, including RSS/Atom feeds, publisher APIs, and webhooks.
- Normalization & Storage: Article content and metadata (title, body/snippet, author, publish time, canonical URL, source, topics/tags) need to be normalized and stored efficiently.
- De-duplication: Identify and de-duplicate near-identical stories that may appear across multiple sources, often reporting on the same event.
- Ranking & Serving:
- Provide a global "homepage" feed with universally ranked stories.
- Offer topic-specific feeds (e.g., Sports, Technology).
- Optionally, support personalized feeds tailored to individual user interests.
- Performance: Ensure low-latency reads for feed browsing, with a strong emphasis on freshness, meaning new stories should appear quickly.
Non-Functional Requirements
- Availability: The system must be highly available, supporting multi-region reads.
- Scalability: It should be capable of handling significant traffic spikes, particularly during breaking news events.
- Content Safety: Implement reasonable measures for basic spam detection and handling of malicious sources.
Scope of Discussion
Candidates should cover the following architectural aspects, making reasonable assumptions about traffic, QPS, and data volume as needed:
- API design for both article ingestion and feed consumption.
- Data model and choices for persistent storage.
- The complete ingestion and processing pipeline, including parsing, content enrichment, and de-duplication.
- Approaches to ranking, considering various signals and the balance between batch and real-time processing.
- Strategies for caching and efficient feed generation.
- Considerations for system reliability, handling backfills, and comprehensive monitoring.