Design email to avoid Promotions without online tests
Offline Design of a Transactional Email to Minimize Promotions/Spam Classification
Context
You are tasked with finalizing the design of a single in-game transactional email before it is sent. The primary objective is to minimize the probability that mailbox providers (such as Gmail, Outlook, and Yahoo) classify this email as "Promotions" or "Spam." A critical constraint is that no online A/B tests or post-send user feedback can be utilized; the design decision must be made entirely offline based on historical data.
Available Data
You have access to 12 months of historical email data, including the following fields:
send_id: Unique identifier for each email send.subject: The subject line of the email.body_html: The HTML content of the email body.num_links: The total number of links within the email.link_domains: A list of domains linked in the email (e.g.,game.com,store.game.com,help.game.com,partner.com).anchor_text_types: Classification of anchor text styles (e.g., "CTA" for Call-to-Action vs. "neutral").num_images: The number of images present in the email.template_id: Identifier for the email template used.sender_reputation_metrics: Includescomplaint_rate_7d(7-day complaint rate),bounce_rate_7d(7-day bounce rate), anddomain_dkim_pass(DKIM pass status for the domain).send_time_utc: The UTC timestamp of the email send.segment: Categorization by region and platform.provider: The mailbox provider (e.g., "gmail", "outlook", "yahoo").label: The classification from seed inboxing tests or logs, indicatingfolder_labelin{primary, promotions, spam}.
Design Variables
You have control over the following design variables for the new email:
- Number of links (constrained between 1 and 5).
- Which specific domains are linked.
- Anchor text style (either CTA or neutral).
- Subject tokens (subject length must be less than or equal to 60 characters).
- Presence of a hero image (binary choice).
Business Constraints
- The email must include at least one link to
help.game.com. - If the predicted risk associated with including
partner.comlinks exceeds a predefined threshold,partner.comlinks must not be included.
Tasks
a. Problem Formulation: Formulate an offline risk-minimization problem to select the optimal email design (represented as a decision vector). This formulation should include the objective function, all relevant constraints, and any robustness terms (e.g., worst-case risk across providers or conformal upper bounds).
b. Feature Engineering and Modeling: Specify the features to be used and propose a suitable modeling approach to estimate the risk. Detail strategies to prevent data leakage (e.g., time-based data splits, appropriate handling of template_id) and how to ensure the probabilities are well-calibrated.
c. Covariate Shift Mitigation: Describe how you would address covariate shift between the historical email templates and the proposed new design. Consider techniques such as domain adaptation, imposing monotonic constraints on variables like num_links, or generating semi-synthetic data.
d. Search and Optimization Strategy: Propose a strategy for searching and optimizing over the discrete design space. Examples include beam search with a learned surrogate model, Bayesian optimization for mixed variables, or Integer Linear Programming (ILP) with a learned risk function.
e. Offline Validation: Explain how to validate the chosen design entirely offline without any new user feedback. Discuss methods like off-policy evaluation with inverse propensity weighting, stratified provider-wise risk assessment, and conformal prediction intervals.
f. Scarce/Noisy Labels Fallback: If historical labels are scarce or noisy for certain providers, propose a fallback strategy. This could involve weak labeling using an open-source classifier combined with a small hand-labeled dataset. Additionally, explain how you would quantify the added uncertainty introduced by this fallback in the final decision.
g. Decision Rule: Deliver a concrete decision rule for selecting the final email design (e.g., "select the lowest-risk design whose worst-case provider risk upper bound at 90% confidence is below X%"). Justify your choice for the threshold 'X'.
AI interview prep
Core dilemma
Making a critical, high-stakes decision (email design impacting deliverability) without any direct online feedback or A/B testing, relying solely on historical data and robust offline evaluation under conditions of potential covariate shift and label scarcity.
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Design email to avoid Promotions without online tests
Offline Design of a Transactional Email to Minimize Promotions/Spam Classification
Context
You are tasked with finalizing the design of a single in-game transactional email before it is sent. The primary objective is to minimize the probability that mailbox providers (such as Gmail, Outlook, and Yahoo) classify this email as "Promotions" or "Spam." A critical constraint is that no online A/B tests or post-send user feedback can be utilized; the design decision must be made entirely offline based on historical data.
Available Data
You have access to 12 months of historical email data, including the following fields:
send_id: Unique identifier for each email send.subject: The subject line of the email.body_html: The HTML content of the email body.num_links: The total number of links within the email.link_domains: A list of domains linked in the email (e.g.,game.com,store.game.com,help.game.com,partner.com).anchor_text_types: Classification of anchor text styles (e.g., "CTA" for Call-to-Action vs. "neutral").num_images: The number of images present in the email.template_id: Identifier for the email template used.sender_reputation_metrics: Includescomplaint_rate_7d(7-day complaint rate),bounce_rate_7d(7-day bounce rate), anddomain_dkim_pass(DKIM pass status for the domain).send_time_utc: The UTC timestamp of the email send.segment: Categorization by region and platform.provider: The mailbox provider (e.g., "gmail", "outlook", "yahoo").label: The classification from seed inboxing tests or logs, indicatingfolder_labelin{primary, promotions, spam}.
Design Variables
You have control over the following design variables for the new email:
- Number of links (constrained between 1 and 5).
- Which specific domains are linked.
- Anchor text style (either CTA or neutral).
- Subject tokens (subject length must be less than or equal to 60 characters).
- Presence of a hero image (binary choice).
Business Constraints
- The email must include at least one link to
help.game.com. - If the predicted risk associated with including
partner.comlinks exceeds a predefined threshold,partner.comlinks must not be included.
Tasks
a. Problem Formulation: Formulate an offline risk-minimization problem to select the optimal email design (represented as a decision vector). This formulation should include the objective function, all relevant constraints, and any robustness terms (e.g., worst-case risk across providers or conformal upper bounds).
b. Feature Engineering and Modeling: Specify the features to be used and propose a suitable modeling approach to estimate the risk. Detail strategies to prevent data leakage (e.g., time-based data splits, appropriate handling of template_id) and how to ensure the probabilities are well-calibrated.
c. Covariate Shift Mitigation: Describe how you would address covariate shift between the historical email templates and the proposed new design. Consider techniques such as domain adaptation, imposing monotonic constraints on variables like num_links, or generating semi-synthetic data.
d. Search and Optimization Strategy: Propose a strategy for searching and optimizing over the discrete design space. Examples include beam search with a learned surrogate model, Bayesian optimization for mixed variables, or Integer Linear Programming (ILP) with a learned risk function.
e. Offline Validation: Explain how to validate the chosen design entirely offline without any new user feedback. Discuss methods like off-policy evaluation with inverse propensity weighting, stratified provider-wise risk assessment, and conformal prediction intervals.
f. Scarce/Noisy Labels Fallback: If historical labels are scarce or noisy for certain providers, propose a fallback strategy. This could involve weak labeling using an open-source classifier combined with a small hand-labeled dataset. Additionally, explain how you would quantify the added uncertainty introduced by this fallback in the final decision.
g. Decision Rule: Deliver a concrete decision rule for selecting the final email design (e.g., "select the lowest-risk design whose worst-case provider risk upper bound at 90% confidence is below X%"). Justify your choice for the threshold 'X'.