Choose KPIs and prove impact with experiments
Expedia Search: Profit-Focused Ranking Model Launch
Context
You are a key member of the Search team, tasked with launching a new hotel-ranking model. This model is specifically designed for enterprise clients who prioritize profitable bookings over simple clicks. For attribution purposes, assume a 7-day booking window from the initial search event. It's also known that cancellations typically occur within 30 days of booking, and you have the capability to estimate an expected show-up probability at the time of booking.
Tasks
- (a) Define Primary KPI & Guardrail Metrics: Define a single primary Key Performance Indicator (KPI) with an exact formula that directly aligns with client value (e.g., margin-adjusted bookings per search within a 7-day attribution window). Additionally, list at least three guardrail metrics, each with explicit numeric thresholds (e.g., cancellation rate ≤ X%, latency p95 ≤ Y ms, diversity entropy ≥ Z).
- (b) Validate Offline Proxy Metrics: Explain your methodology for validating that offline proxy metrics (e.g., NDCG@10 weighted by booking margin) accurately correlate with the online primary KPI. Specify the analysis design, acceptable correlation or elasticity ranges, and the actions you would take if a misalignment is detected.
- (c) Design an A/B Test: Design a comprehensive A/B test for the new model. This design should include: your chosen randomization unit (e.g., search session), the traffic split strategy, a ramp strategy for deployment, the calculation of Minimum Detectable Effect (MDE) and required sample size (stating your base rates and variance assumptions), essential pre-experiment checks (e.g., Sample Ratio Mismatch - SRM), and proposed variance reduction techniques (e.g., CUPED, stratification).
- (d) Detect & Prevent Metric Gaming/Novelty Effects: Describe how you would detect and proactively prevent issues such as metric gaming (e.g., clickbait, artificial position churn) and novelty effects. Also, define a clear criterion for when a rollback of the new model would be necessary.
- (e) Ship or Rollback Decision: Consider a scenario where, post-launch, bookings remain flat but cancellations in the treatment group rise by 15%. Decide whether to ship the model or roll it back, providing a detailed justification based on the expected profit impact and associated confidence intervals.
- (f) Estimate Incremental Gross Profit: Outline how you would estimate the incremental gross profit per search, including a 95% confidence interval. Describe your approach to separating this incrementality from mere attribution.
- (g) Multi-Touch Attribution for Paid Channels: For paid channels, propose a multi-touch attribution (MTA) approach (e.g., comparing Shapley/Markov models against regression-based models or geo-experiments). Explain how you would calibrate your chosen MTA model against holdout or geo tests.
AI interview prep
Core dilemma
Accurately measuring and attributing true business profit in a complex, multi-stage user journey (search -> booking -> show-up), while designing robust experiments, preventing metric manipulation, and ensuring offline proxies reliably predict online value.
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Choose KPIs and prove impact with experiments
Expedia Search: Profit-Focused Ranking Model Launch
Context
You are a key member of the Search team, tasked with launching a new hotel-ranking model. This model is specifically designed for enterprise clients who prioritize profitable bookings over simple clicks. For attribution purposes, assume a 7-day booking window from the initial search event. It's also known that cancellations typically occur within 30 days of booking, and you have the capability to estimate an expected show-up probability at the time of booking.
Tasks
- (a) Define Primary KPI & Guardrail Metrics: Define a single primary Key Performance Indicator (KPI) with an exact formula that directly aligns with client value (e.g., margin-adjusted bookings per search within a 7-day attribution window). Additionally, list at least three guardrail metrics, each with explicit numeric thresholds (e.g., cancellation rate ≤ X%, latency p95 ≤ Y ms, diversity entropy ≥ Z).
- (b) Validate Offline Proxy Metrics: Explain your methodology for validating that offline proxy metrics (e.g., NDCG@10 weighted by booking margin) accurately correlate with the online primary KPI. Specify the analysis design, acceptable correlation or elasticity ranges, and the actions you would take if a misalignment is detected.
- (c) Design an A/B Test: Design a comprehensive A/B test for the new model. This design should include: your chosen randomization unit (e.g., search session), the traffic split strategy, a ramp strategy for deployment, the calculation of Minimum Detectable Effect (MDE) and required sample size (stating your base rates and variance assumptions), essential pre-experiment checks (e.g., Sample Ratio Mismatch - SRM), and proposed variance reduction techniques (e.g., CUPED, stratification).
- (d) Detect & Prevent Metric Gaming/Novelty Effects: Describe how you would detect and proactively prevent issues such as metric gaming (e.g., clickbait, artificial position churn) and novelty effects. Also, define a clear criterion for when a rollback of the new model would be necessary.
- (e) Ship or Rollback Decision: Consider a scenario where, post-launch, bookings remain flat but cancellations in the treatment group rise by 15%. Decide whether to ship the model or roll it back, providing a detailed justification based on the expected profit impact and associated confidence intervals.
- (f) Estimate Incremental Gross Profit: Outline how you would estimate the incremental gross profit per search, including a 95% confidence interval. Describe your approach to separating this incrementality from mere attribution.
- (g) Multi-Touch Attribution for Paid Channels: For paid channels, propose a multi-touch attribution (MTA) approach (e.g., comparing Shapley/Markov models against regression-based models or geo-experiments). Explain how you would calibrate your chosen MTA model against holdout or geo tests.