Blogs›Instacart›Experimentation Variance Reduction
Introduced a novel method for variance reduction in experiments by building predictions at a finer grain (order-level) and aggregating them up, significantly improving statistical power and reducing experimentation time. This approach leverages order-specific covariates to create more accurate pre-treatment predictions, overcoming limitations of traditional region-day CUPED. This post further details the application of fixed effects models, specifically addressing the computational challenges of high-cardinality fixed effects in marketplace modeling using PyFixest and the Frisch-Waugh-Lovell theorem to improve processing speed, memory efficiency, and estimator precision.
7/1/2026
Developed and implemented an 'Order-Level CUPED' methodology that trains an order-level prediction model using pre-treatment features. These predictions are then aggregated to the region-day grain, serving as the covariate in the CUPED adjustment. This method demonstrated a 18-40% variance reduction, leading to an average one-third reduction in experiment runtimes, compared to standard region-day CUPED.
6/29/2026
This post details the application of fixed effects models for high-cardinality marketplace modeling, addressing the computational bottleneck of traditional OLS regression when dealing with a large number of categories. It explains how the Frisch-Waugh-Lovell theorem and modern packages like PyFixest bypass these limitations by demeaning data and effectively reducing the size of the Gram matrix. The post also includes benchmarking to demonstrate the real-world impact on processing speed, memory efficiency, and estimator precision.