Design Experiments Under Network Interference
Experimentation Design for a Gaming Platform
You are a Senior Data Scientist for a major gaming platform, such as PlayStation. You are tasked with addressing two critical areas related to experimentation and A/B testing.
1. A/A Test Validation
The platform's experimentation system recently ran an A/A test designed to validate its randomization and logging mechanisms. The intended split for control and treatment groups was 50% each. However, the observed assignment ratio was 51% for the control group and 49% for the treatment group.
- Is this observed imbalance a problem?
- How would you rigorously test whether this imbalance is statistically meaningful?
- Assuming the imbalance is statistically meaningful, what potential root causes would you investigate thoroughly before confidently trusting the results of any future A/B tests conducted on this platform?
2. Experiment Design with Social Network Effects
The product team is planning to launch a new feature that enables friends to send virtual gifts to each other within the game. The primary hypothesis is that this gifting functionality will lead to an increase in overall user engagement.
- Define the primary metric that would directly measure the success of this feature, along with relevant guardrail metrics and clear success criteria.
- Outline a detailed experimental design to accurately estimate the causal impact of this new gifting feature.
- When considering the randomization unit for this experiment, which would you choose among user-level, friend-cluster-level, region-level, or an alternative approach? Justify your choice.
- How would your proposed experimental design need to adapt or change if the users' friendship graph is highly connected, meaning almost every user is directly or indirectly linked to almost every other user?
- What specific biases or validity threats would you anticipate and worry about with your chosen design, especially given the social nature of the feature?
AI interview prep
Core dilemma
Ensuring the statistical validity and causal interpretability of A/B tests in a large-scale, interconnected user environment, balancing experimental rigor against practical implementation challenges and the pervasive influence of network effects.
2 tradeoffs and 2 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Design Experiments Under Network Interference
Experimentation Design for a Gaming Platform
You are a Senior Data Scientist for a major gaming platform, such as PlayStation. You are tasked with addressing two critical areas related to experimentation and A/B testing.
1. A/A Test Validation
The platform's experimentation system recently ran an A/A test designed to validate its randomization and logging mechanisms. The intended split for control and treatment groups was 50% each. However, the observed assignment ratio was 51% for the control group and 49% for the treatment group.
- Is this observed imbalance a problem?
- How would you rigorously test whether this imbalance is statistically meaningful?
- Assuming the imbalance is statistically meaningful, what potential root causes would you investigate thoroughly before confidently trusting the results of any future A/B tests conducted on this platform?
2. Experiment Design with Social Network Effects
The product team is planning to launch a new feature that enables friends to send virtual gifts to each other within the game. The primary hypothesis is that this gifting functionality will lead to an increase in overall user engagement.
- Define the primary metric that would directly measure the success of this feature, along with relevant guardrail metrics and clear success criteria.
- Outline a detailed experimental design to accurately estimate the causal impact of this new gifting feature.
- When considering the randomization unit for this experiment, which would you choose among user-level, friend-cluster-level, region-level, or an alternative approach? Justify your choice.
- How would your proposed experimental design need to adapt or change if the users' friendship graph is highly connected, meaning almost every user is directly or indirectly linked to almost every other user?
- What specific biases or validity threats would you anticipate and worry about with your chosen design, especially given the social nature of the feature?