Beyond A/B Testing: Using Surrogacy and Region-Splits to Measure Long-Term Effects in Marketplaces

What this post added

This post details a framework for estimating long-term effects of marketplace decisions, specifically focusing on 'market-mediated long-term effects'. It introduces a two-step approach: 1) estimating how policy changes affect negative user experiences (e.g., long waits, high surge) using residualized regressions and validating with switch-back experiments, and 2) estimating how these negative experiences affect future user behavior using double-robust causal inference (AIPW) and validating with user-split experiments. The overall long-term effects are then verified using region-split experiments, with a forward selection algorithm for optimizing region selection.