Evaluate Metrics and Randomization for Onboarding Tutorial Change
Scenario
A specific step within Confluent's multi-step user onboarding tutorial has been modified. The product team aims to conduct an experiment to evaluate whether this change improves the user experience at that particular step, while rigorously ensuring no adverse impact on the overall onboarding flow.
Assumptions
- The tutorial consists of a sequence of ordered steps (1 through k). Only step 'i' has been altered; all other steps remain unchanged.
- We have the capability to instrument and track various events at the step level, including:
step_i_view,step_i_submit,step_i_success,step_i_error,help_click,backtrack,abandon, and associatedtimestamps. - Users can be associated with accounts (organizations) that may contain multiple users.
Questions
- Metrics: Identify the primary and secondary metrics that would be most relevant and specific to tracking the impact of the modified step.
- Experiment Design:
- Determine the appropriate level of randomization (user-level vs. account-level). Justify your choice.
- What covariates would you examine to verify the comparability of your experimental groups?
- Inference and Sizing:
- Which statistical test(s) would you employ for analysis?
- Describe how you would compute the required sample size and estimate the expected runtime for the experiment.
- If the resulting sample size turns out to be very small, what alternative statistical test would you prefer and why?
AI interview prep
Core dilemma
Ensuring valid statistical inference and avoiding contamination in an A/B test within a multi-user, multi-step sequential flow, while balancing sensitivity to micro-conversions and robustness to overall funnel impact.
3 tradeoffs and 2 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Evaluate Metrics and Randomization for Onboarding Tutorial Change
Scenario
A specific step within Confluent's multi-step user onboarding tutorial has been modified. The product team aims to conduct an experiment to evaluate whether this change improves the user experience at that particular step, while rigorously ensuring no adverse impact on the overall onboarding flow.
Assumptions
- The tutorial consists of a sequence of ordered steps (1 through k). Only step 'i' has been altered; all other steps remain unchanged.
- We have the capability to instrument and track various events at the step level, including:
step_i_view,step_i_submit,step_i_success,step_i_error,help_click,backtrack,abandon, and associatedtimestamps. - Users can be associated with accounts (organizations) that may contain multiple users.
Questions
- Metrics: Identify the primary and secondary metrics that would be most relevant and specific to tracking the impact of the modified step.
- Experiment Design:
- Determine the appropriate level of randomization (user-level vs. account-level). Justify your choice.
- What covariates would you examine to verify the comparability of your experimental groups?
- Inference and Sizing:
- Which statistical test(s) would you employ for analysis?
- Describe how you would compute the required sample size and estimate the expected runtime for the experiment.
- If the resulting sample size turns out to be very small, what alternative statistical test would you prefer and why?