Sign in to track progress, unlock prep layers, and browse your pipeline.Sign in
Design real-time live-stream recommendations
Design a Real-Time Recommendation System for Live Streams
This problem focuses on designing a real-time recommendation system for a large live-streaming platform, similar to Twitch. You should assume access to standard logs, including impressions, clicks, view events with join/leave timestamps, and comprehensive stream metadata.
Address the following six key areas:
1. Objective and Labels
- Define the primary objective of your recommendation system (e.g., maximize post-click 10-minute watch probability).
- Detail how to construct training labels from view events, specifically addressing the handling of delayed outcomes and right-censoring when a stream concludes before the defined label horizon.
2. Features
- List relevant user, creator, stream, and context features.
- Explain strategies for handling the cold-start problem for both new creators and new users.
- Describe how to effectively incorporate short-term session signals and long-term embeddings into your feature set.
3. Model and Loss
- Choose an appropriate model family (e.g., a two-tower retrieval model combined with a re-ranker, designed for calibrated probabilities).
- Specify the loss function(s), your negative sampling strategy (including in-batch negatives and hard negatives), and how you would address severe class imbalance.
4. Feedback Loops and Bias
- Propose methods to mitigate position bias and popularity bias, such as using counterfactual estimation (e.g., Inverse Propensity Scoring (IPS) or Doubly Robust (DR) estimators) or randomized exploration.
- Describe your exploration policy (e.g., Thompson Sampling or Upper Confidence Bound (UCB)) and outline necessary safety constraints.
5. Evaluation
- Define a set of offline metrics (e.g., AUC, PR-AUC, calibration, NDCG@k) and online metrics (e.g., watch time per viewer, session length, bounce rate).
- Explain how to establish a reliable correlation between offline and online performance using techniques like replay evaluation and interleaving.
6. Serving and Freshness
- Propose an end-to-end latency budget, targeting less than 100 ms at the 95th percentile (p95), encompassing retrieval, feature fetching, and ranking stages.
- Describe how to ensure feature freshness (e.g., detecting when a streamer goes live) and how you would update embeddings in near real-time.
AI interview prep
Core dilemma
Balancing the extreme demands of real-time data freshness and ultra-low serving latency (<100ms p95) with the complexity of sophisticated ML models, robust bias mitigation, and effective exploration strategies at Twitch's scale.
Mid-level signals4
Senior signals5
Red flags5
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Design real-time live-stream recommendations
Design a Real-Time Recommendation System for Live Streams
This problem focuses on designing a real-time recommendation system for a large live-streaming platform, similar to Twitch. You should assume access to standard logs, including impressions, clicks, view events with join/leave timestamps, and comprehensive stream metadata.
Address the following six key areas:
1. Objective and Labels
- Define the primary objective of your recommendation system (e.g., maximize post-click 10-minute watch probability).
- Detail how to construct training labels from view events, specifically addressing the handling of delayed outcomes and right-censoring when a stream concludes before the defined label horizon.
2. Features
- List relevant user, creator, stream, and context features.
- Explain strategies for handling the cold-start problem for both new creators and new users.
- Describe how to effectively incorporate short-term session signals and long-term embeddings into your feature set.
3. Model and Loss
- Choose an appropriate model family (e.g., a two-tower retrieval model combined with a re-ranker, designed for calibrated probabilities).
- Specify the loss function(s), your negative sampling strategy (including in-batch negatives and hard negatives), and how you would address severe class imbalance.
4. Feedback Loops and Bias
- Propose methods to mitigate position bias and popularity bias, such as using counterfactual estimation (e.g., Inverse Propensity Scoring (IPS) or Doubly Robust (DR) estimators) or randomized exploration.
- Describe your exploration policy (e.g., Thompson Sampling or Upper Confidence Bound (UCB)) and outline necessary safety constraints.
5. Evaluation
- Define a set of offline metrics (e.g., AUC, PR-AUC, calibration, NDCG@k) and online metrics (e.g., watch time per viewer, session length, bounce rate).
- Explain how to establish a reliable correlation between offline and online performance using techniques like replay evaluation and interleaving.
6. Serving and Freshness
- Propose an end-to-end latency budget, targeting less than 100 ms at the 95th percentile (p95), encompassing retrieval, feature fetching, and ranking stages.
- Describe how to ensure feature freshness (e.g., detecting when a streamer goes live) and how you would update embeddings in near real-time.