Variance Reduction Below the Randomization Grain

This post details a method for variance reduction in experiments by training an order-level prediction model (e.g., Gradient Boosted Tree) using pre-treatment features, and then aggregating these predictions to the region-day grain. This aggregated prediction is used as a covariate in the CUPED adjustment. The post highlights that this approach can cut variance by 18-40%, significantly more than standard region-day CUPED, leading to an average reduction in experimental runtimes by about a third. It also introduces a placebo test to detect feature contamination by checking if the aggregated prediction's treatment effect is statistically indistinguishable from zero.