Design an ads ranking ML system
Design an Ads Ranking ML System
You are tasked with designing an ads ranking system for a large consumer application, such as a feed or search entry point. For each incoming request, the system will receive user context and a set of eligible ads (candidates). The primary goal is to return a ranked list of ads that maximizes long-term value (e.g., revenue) while ensuring a positive user experience.
Requirements
- Primary Business Goal: Maximize long-term user value (e.g., revenue) while maintaining a good user experience.
- Optimization Targets: Must handle multiple optimization targets, including Click-Through Rate (CTR), Conversion Rate (CVR), and expected revenue.
- Latency Budget: The ranking stage must operate within a latency budget of tens of milliseconds. Candidate generation is assumed to be handled separately.
- ML Emphasis: Strong focus on Machine Learning aspects, including feature design, training data construction, modeling choices, and multi-task learning.
Questions to Address
- Feature Engineering: What features would you design (user, ad, context, cross-features)? How would you generate and maintain their freshness?
- Training Data: How would you define labels and construct training data, considering biases like position bias and selection bias from the existing ranker?
- Ranking Model & Objectives: Propose a ranking model. Explain how you would combine multiple objectives (e.g., CTR, CVR, value). If using multi-task learning, describe the architecture and loss function.
- Evaluation: How would you evaluate the model both offline and online? What key metrics and guardrails would you implement?
- Practical Issues: How would you address practical challenges such as cold start, delayed conversions, distribution shift, and feature leakage?
AI interview prep
Core dilemma
Balancing the complexity of a multi-objective, personalized ML ranking model against strict low-latency serving requirements, while mitigating inherent data biases.
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Design an ads ranking ML system
Design an Ads Ranking ML System
You are tasked with designing an ads ranking system for a large consumer application, such as a feed or search entry point. For each incoming request, the system will receive user context and a set of eligible ads (candidates). The primary goal is to return a ranked list of ads that maximizes long-term value (e.g., revenue) while ensuring a positive user experience.
Requirements
- Primary Business Goal: Maximize long-term user value (e.g., revenue) while maintaining a good user experience.
- Optimization Targets: Must handle multiple optimization targets, including Click-Through Rate (CTR), Conversion Rate (CVR), and expected revenue.
- Latency Budget: The ranking stage must operate within a latency budget of tens of milliseconds. Candidate generation is assumed to be handled separately.
- ML Emphasis: Strong focus on Machine Learning aspects, including feature design, training data construction, modeling choices, and multi-task learning.
Questions to Address
- Feature Engineering: What features would you design (user, ad, context, cross-features)? How would you generate and maintain their freshness?
- Training Data: How would you define labels and construct training data, considering biases like position bias and selection bias from the existing ranker?
- Ranking Model & Objectives: Propose a ranking model. Explain how you would combine multiple objectives (e.g., CTR, CVR, value). If using multi-task learning, describe the architecture and loss function.
- Evaluation: How would you evaluate the model both offline and online? What key metrics and guardrails would you implement?
- Practical Issues: How would you address practical challenges such as cold start, delayed conversions, distribution shift, and feature leakage?