  We study ratio metrics in A/B testing at the presence of correlation among
observations coming from the same user and provides practical guidance
especially when two metrics contradict each other. We propose new estimating
methods to quantitatively measure the intra-user correlation (within segments).
With the accurately estimated correlation, a uniformly minimum-variance
unbiased estimator of the population mean, called correlation-adjusted mean, is
proposed to account for such correlation structure. It is proved theoretically
and numerically better than the other two unbiased estimators, naive mean and
normalized mean (averaging within users first and then across users). The
correlation-adjusted mean method is unbiased and has reduced variance so it
gains additional power. Several simulation studies are designed to show the
estimation accuracy of the correlation structure, effectiveness in reducing
variance, and capability of obtaining more power. An application to the eBay
data is conducted to conclude this paper.
