Bringing Sequential Testing to Experiments with Longitudinal Data (Part 1): The Peeking Problem 2.0

This post introduces the 'peeking problem 2.0,' which arises in sequential testing with longitudinal data (multiple observations per unit). It details the challenges of analyzing open-ended metrics where within-unit peeking can inflate false positive rates, even with sequential tests. The post outlines the need to separate measurements, metrics, estimands, estimators, and statistical tests, and uses potential outcomes notation to precisely define the statistical goal. It also includes a Monte Carlo simulation study demonstrating the inflated FPRs of standard sequential tests on open-ended metrics.