AB实验的Variance Reduction

Brief

• Attempt to understand the inherent uncertainty introduced by individual experimental units.
• Estimate the pre-experiment individual variance for each unit.
• For examle, if the target metric is "time spent by someone on the site per day", we could estimate the variance of a person's daily tiem spent during the month before the experiment and give more weight to those who previously exhibited lower variance (less noisy behaviors).

Outcome

average variance reduction of 17 percent, while bias is bounded with 2 percent

Implementation

estimate the variance for each unit

     unpooled estimators (using the pre-experiment user-level sampling variance), building a machine learning model to predict out-of-sample variance from features, and using Empirical-Bayes estimators to pool information across those using our platform.

Reference

Increasing the sensitivity of A/B tests by utilizing the variance estimates of experimental units