How to Accurately Test Significance with Difference in Difference Models
Sep 1, 2023Bradley Doll (Insights Manager), Jeffrey Li (Insights Manager), and Tolga Oztan (Sr. Data Scientist)
Summary
This post introduces and evaluates three methods (averaging, clustering, and permutation testing) for accurately estimating significance in difference-in-differences (DID) models when dealing with autocorrelated time series data. It highlights the limitations of standard OLS significance tests in such scenarios and provides simulation-based evidence on the trade-offs between power and false positives for each method, recommending permutation testing for smaller datasets and clustered standard errors for larger ones.
More from the Experimentation Platform trail
- Why Spotify Is Not Using Bayesian A/B Testing | Spotify Engineering
- When Can LLMs Replace Humans in A/B Tests? | Spotify Engineering
- Better Experiments with LLM Evals — A funnel, not a fork | Spotify Engineering
- Why We Use Separate Tech Stacks for Personalization and Experimentation
- Beyond Winning: Spotify’s Experiments with Learning Framework