Evaluate Overlapping Shelf Ranking Experiments
Product Context
Fetch operates a mobile application that features product shelves. Each shelf contains multiple products, and the application also determines the order in which these shelves are presented to users.
Experiments
Fetch is currently running two concurrent A/B experiments within its mobile app:
-
Experiment 1: Product Ordering within a Shelf
- Treatment: Modifies the display order of products inside each shelf.
- Traffic Allocation: 50% of the app's total traffic is exposed to this experiment.
-
Experiment 2: Shelf Ordering in the App
- Treatment: Modifies the display order of shelves shown within the app.
- Traffic Allocation: 9% of the app's total traffic is exposed to this experiment.
Business Goal
The primary objective for these experiments is to enhance user engagement and shopping behavior. Key performance indicators include, but are not limited to, product clicks, add-to-cart events, purchases, revenue generated, and receipt submissions.
Questions
- What critical factors should be considered when designing and analyzing these two A/B experiments?
- If perfect isolation of traffic between the two experiments is not achievable, does this render the experiment results entirely unusable? Justify your reasoning.
- Assuming the two experiments run with overlapping traffic, how would you quantify and measure the specific impact of Experiment 2 on Experiment 1?
- What specific metrics, guardrails, statistical checks, and potential interpretation risks would you incorporate into your comprehensive analysis?
AI interview prep
Core dilemma
Accurately attributing causal impact and understanding interaction effects when running multiple, potentially overlapping A/B experiments on the same user population and UI elements, while maintaining statistical validity and business velocity.
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Evaluate Overlapping Shelf Ranking Experiments
Product Context
Fetch operates a mobile application that features product shelves. Each shelf contains multiple products, and the application also determines the order in which these shelves are presented to users.
Experiments
Fetch is currently running two concurrent A/B experiments within its mobile app:
-
Experiment 1: Product Ordering within a Shelf
- Treatment: Modifies the display order of products inside each shelf.
- Traffic Allocation: 50% of the app's total traffic is exposed to this experiment.
-
Experiment 2: Shelf Ordering in the App
- Treatment: Modifies the display order of shelves shown within the app.
- Traffic Allocation: 9% of the app's total traffic is exposed to this experiment.
Business Goal
The primary objective for these experiments is to enhance user engagement and shopping behavior. Key performance indicators include, but are not limited to, product clicks, add-to-cart events, purchases, revenue generated, and receipt submissions.
Questions
- What critical factors should be considered when designing and analyzing these two A/B experiments?
- If perfect isolation of traffic between the two experiments is not achievable, does this render the experiment results entirely unusable? Justify your reasoning.
- Assuming the two experiments run with overlapping traffic, how would you quantify and measure the specific impact of Experiment 2 on Experiment 1?
- What specific metrics, guardrails, statistical checks, and potential interpretation risks would you incorporate into your comprehensive analysis?