  Online controlled experiments play a crucial role in enabling data-driven
decisions across a wide range of companies. Variance reduction is an effective
technique to improve the sensitivity of experiments, achieving higher
statistical power while using fewer samples and shorter experimental periods.
However, typical variance reduction methods (e.g., regression-adjusted
estimators) are built upon the intuitional assumption of Gaussian distributions
and cannot properly characterize the real business metrics with heavy-tailed
distributions. Furthermore, outliers diminish the correlation between
pre-experiment covariates and outcome metrics, greatly limiting the
effectiveness of variance reduction.
  In this paper, we develop a novel framework that integrates the Student's
t-distribution with machine learning tools to fit heavy-tailed metrics and
construct a robust average treatment effect estimator in online controlled
experiments, which we call STATE. By adopting a variational EM method to
optimize the loglikehood function, we can infer a robust solution that greatly
eliminates the negative impact of outliers and achieves significant variance
reduction. Moreover, we extend the STATE method from count metrics to ratio
metrics by utilizing linear transformation that preserves unbiased estimation,
whose variance reduction is more complex but less investigated in existing
works. Finally, both simulations on synthetic data and long-term empirical
results on Meituan experiment platform demonstrate the effectiveness of our
method. Compared with the state-of-the-art estimators (CUPAC/MLRATE), STATE
achieves over 50% variance reduction, indicating it can reach the same
statistical power with only half of the observations, or half the experimental
duration.
