  We describe how to calculate standard errors for A/B tests that include
clustered data, ratio metrics, and/or covariate adjustment. We may do this for
power analysis/sample size calculations prior to running an experiment using
historical data, or after an experiment for hypothesis testing and confidence
intervals. The different applications have a common framework, using the sample
variance of certain residuals. The framework is compatible with modular
software, can be plugged into standard tools, doesn't require computing
covariance matrices, and is numerically stable. Using this approach we estimate
that covariate adjustment gives a median 66% variance reduction for a key
metric, reducing experiment run time by 66%.
