TechPrufer
Get started
Instacart
Instacart logoInstacart · Feature trail

Experimentation Variance Reduction

2 posts · Jun 2026 – Jul 2026

What this trail covers

Instacart is enhancing its experimentation capabilities by developing novel methods for variance reduction. This includes building predictions at a finer grain (order-level) and aggregating them up, significantly improving statistical power and reducing experimentation time. This approach leverages order-specific covariates to create more accurate pre-treatment predictions, overcoming limitations of traditional region-day CUPED. The system also incorporates a placebo test to ensure feature contamination does not introduce bias. This has led to substantial reductions in experimental runtimes, allowing for more experiments per quarter and faster identification of marketplace improvements.

All Instacart trails
  1. Variance Reduction Below the Randomization GrainThis post details a method for variance reduction in experiments by training an order-level prediction model (e.g., Gradient Boosted Tree) using pre-treatment features, and then aggregating these predictions to the region-day grain. This aggregated prediction is used as a covariate in the CUPED adjustment. The post highlights that this approach can cut variance by 18-40%, significantly more than standard region-day CUPED, leading to an average reduction in experimental runtimes by about a third. It also introduces a placebo test to detect feature contamination by checking if the aggregated prediction's treatment effect is statistically indistinguishable from zero.
  2. Leveraging PyFixest for High-Cardinality Marketplace Modeling at InstacartThis post details the application of fixed effects models for high-cardinality marketplace modeling, addressing the computational bottleneck of traditional OLS regression when dealing with a large number of categories. It explains how the Frisch-Waugh-Lovell theorem and modern packages like PyFixest bypass these limitations by demeaning data and effectively reducing the size of the Gram matrix. The post also includes benchmarking to demonstrate the real-world impact on processing speed, memory efficiency, and estimator precision.