Design a CVR model for RTB bidding
Problem: Design a CVR Model for Real-Time Bidding (RTB)
You are a data scientist working at a Demand-Side Platform (DSP) involved in Real-Time Bidding (RTB). For every ad opportunity (impression), your system is responsible for making rapid decisions: whether to bid, how much to bid, and which creative (ad) to display.
Your primary task is to design a machine learning approach to predict the conversion probability (CVR) for a specific advertising campaign (e.g., "Nike shoes").
Available Historical Data
You have access to the following historical data:
- Impressions Table: Contains
impression_id, timestamp, user/device/context information, publisher/app/site details, geographical data, auction metadata, bid price, and whether the impression resulted in a win or loss. - Clicks Table: Includes
impression_idand click timestamp. This data is optional and often sparse. - Conversions Table: Contains
impression_id(or a user-level attribution key) and conversion timestamp/value. This data is typically very sparse.
Core Tasks
- Define the Learning Target: Clearly articulate what "predict conversion over impression" signifies in this context. This includes selecting an appropriate attribution window and defining the label for your model.
- Feature Engineering: Propose a realistic set of features that would be available at bid time in an RTB environment. Discuss potential risks of data leakage.
- Model Selection: Choose a baseline machine learning model and a candidate model for production (e.g., Logistic Regression vs. Gradient-Boosted Trees like LightGBM). Explain the tradeoffs between these choices.
- Loss Function Justification: Select an appropriate loss function and provide a clear justification for its use. Explain why log loss / binary cross-entropy would be preferred over metrics like Mean Squared Error (MSE) or Area Under the ROC Curve (AUC) as a training loss.
- Handling Class Imbalance: Given that conversions are rare events, describe at least two distinct methods to address class imbalance (e.g., class weighting, negative downsampling). Discuss how these choices impact probability calibration.
- Offline Evaluation: Define a comprehensive offline evaluation strategy, including appropriate metrics and a validation scheme suitable for a non-stationary ad-tech environment. Explain why Precision-Recall AUC (PR-AUC) might be more informative than Receiver Operating Characteristic AUC (ROC-AUC) in this scenario.
Assumptions and Constraints
- Inference Latency: Strict latency constraints apply at inference time, requiring predictions within tens of milliseconds.
- Downstream Usage: The predicted probabilities will be directly utilized downstream for making real-time bidding decisions.
AI interview prep
Core dilemma
Predicting rare, high-value events (conversions) with extreme accuracy and calibration under severe real-time latency constraints in a highly dynamic, non-stationary environment, where mispredictions have direct financial consequences for bidding strategies.
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Design a CVR model for RTB bidding
Problem: Design a CVR Model for Real-Time Bidding (RTB)
You are a data scientist working at a Demand-Side Platform (DSP) involved in Real-Time Bidding (RTB). For every ad opportunity (impression), your system is responsible for making rapid decisions: whether to bid, how much to bid, and which creative (ad) to display.
Your primary task is to design a machine learning approach to predict the conversion probability (CVR) for a specific advertising campaign (e.g., "Nike shoes").
Available Historical Data
You have access to the following historical data:
- Impressions Table: Contains
impression_id, timestamp, user/device/context information, publisher/app/site details, geographical data, auction metadata, bid price, and whether the impression resulted in a win or loss. - Clicks Table: Includes
impression_idand click timestamp. This data is optional and often sparse. - Conversions Table: Contains
impression_id(or a user-level attribution key) and conversion timestamp/value. This data is typically very sparse.
Core Tasks
- Define the Learning Target: Clearly articulate what "predict conversion over impression" signifies in this context. This includes selecting an appropriate attribution window and defining the label for your model.
- Feature Engineering: Propose a realistic set of features that would be available at bid time in an RTB environment. Discuss potential risks of data leakage.
- Model Selection: Choose a baseline machine learning model and a candidate model for production (e.g., Logistic Regression vs. Gradient-Boosted Trees like LightGBM). Explain the tradeoffs between these choices.
- Loss Function Justification: Select an appropriate loss function and provide a clear justification for its use. Explain why log loss / binary cross-entropy would be preferred over metrics like Mean Squared Error (MSE) or Area Under the ROC Curve (AUC) as a training loss.
- Handling Class Imbalance: Given that conversions are rare events, describe at least two distinct methods to address class imbalance (e.g., class weighting, negative downsampling). Discuss how these choices impact probability calibration.
- Offline Evaluation: Define a comprehensive offline evaluation strategy, including appropriate metrics and a validation scheme suitable for a non-stationary ad-tech environment. Explain why Precision-Recall AUC (PR-AUC) might be more informative than Receiver Operating Characteristic AUC (ROC-AUC) in this scenario.
Assumptions and Constraints
- Inference Latency: Strict latency constraints apply at inference time, requiring predictions within tens of milliseconds.
- Downstream Usage: The predicted probabilities will be directly utilized downstream for making real-time bidding decisions.