Design a Trending Articles Platform

Context

Design a backend system capable of surfacing trending articles in near real-time for a large-scale consumer product. The platform must support global, regional (country/locale), and category-specific feeds, and be able to scale to handle hundreds of thousands of events per second at peak.

Your task is to specify the system's requirements and constraints, and then propose a comprehensive end-to-end architecture.

Functional Requirements

• Feed Surfaces and APIs:
  • Provide global, local (country/region), and category-specific feeds.
  • Consider optional personalization versus purely global ranking.
  • Support cursor-based pagination with defined page size limits and stable ordering.
  • Implement language and locale support (e.g., en-US) with appropriate fallback behavior.
• Freshness and Latency:
  • Define a target for feed freshness (e.g., new trends visible within X seconds).
  • Establish Read Latency SLAs (P50/P95/P99).
• Abuse and Spam Resilience:
  • Incorporate mechanisms for bot and fraud mitigation, event deduplication, rate limiting, and downweighting of suspicious activity.
  • Include support for editorial blocks and safety filters.
• Observability and Controls:
  • Design for feature flags, explainability of ranking algorithms, and comprehensive metrics and alerting.

Constraints

As part of your design, you must define and state your assumptions for the following constraints:

• Scale Assumptions:
  • Daily Active Users (DAU), sessions per user per day.
  • Feed Queries Per Second (QPS) (average and peak).
  • Event rates for user actions (views, clicks, dwell time, shares).
  • Peak write rates to caches and data stores.
• SLOs and Availability:
  • P99 read latency target.
  • Freshness and end-to-end pipeline delay (P99).
  • Availability target (e.g., 99.9% or higher).
• Data Retention:
  • Define hot, warm, and cold data retention periods.

Deliverables

Your design should cover the following architectural components and considerations:

1. Ingestion and Storage Architecture for User and Content Events:
   • Design event schemas and enrichment strategies for events like views, clicks, dwell time, and shares.
   • Propose hot, warm, and cold storage tiers with appropriate partitioning strategies.
   • Define an indexing strategy for efficient lookups by article, locale, and category.
2. Streaming and Batch Compute:
   • Implement sliding windows and time-decayed counts for trend calculation.
   • Design for Top-K aggregation per segment (global, locale, category) and merging strategies.
   • Address handling of late/out-of-order events, deduplication, and watermarks.
   • Outline a strategy for backfill and reprocessing of historical data.
3. Caching, Invalidation, and Ranking Service Integration:
   • Detail how to cache and update top lists, including cursor design for pagination.
   • Discuss trade-offs between personalization and global ranking approaches.
4. Consistency Model and Fault Tolerance:
   • Specify whether to use exactly-once or at-least-once guarantees and how to ensure idempotency.
   • Describe failure modes and strategies for graceful degradation.
5. Capacity Planning and Sharding:
   • Provide rough capacity estimates and sharding strategies for key components (e.g., Kafka/stream partitions, cache shards, storage sizing).
   • Ensure sufficient headroom for growth.
6. Technology Choices and Trade-offs:
   • Justify your technology selections (e.g., Kafka vs. Kinesis, Flink vs. Spark, Redis vs. Aerospike, DynamoDB vs. Cassandra, ClickHouse, OpenSearch, S3/Parquet, etc.) based on the defined requirements and constraints.

Presentation Expectations

During the interview, you are expected to verbally describe diagrams (component-by-component), provide API examples, illustrate formulas (e.g., exponential decay), use small numeric examples for capacity planning, and offer clear justifications for all your design decisions.