  In this study, we propose an efficient method for comparing the post-click
metric (e.g., dwell time and conversion rate) of multiple rankings in online
experiments. The proposed method involves (1) the decomposition of the
post-click metric measurement of a ranking into a click model estimation and a
post-click metric measurement of each item in the ranking, and (2) interleaving
of multiple rankings to produce a single ranking that preferentially exposes
items possessing a high population variance. The decomposition of the
post-click metric measurement enables the free layout of items in a ranking and
focuses on the measurement of the post-click metric of each item in the
multiple rankings. The interleaving of multiple rankings reduces the sample
variance of the items possessing a high population variance by optimizing a
ranking to be presented to the users so that those items received more samples
of the post-click metric. In addition, we provide a proof that the proposed
method leads to the minimization of the evaluation error in the ranking
comparison and propose two practical techniques to stabilize the online
experiment. We performed a comprehensive simulation experiment and a real
service setting experiment. The experimental results revealed that (1) the
proposed method outperformed existing methods in terms of efficiency and
accuracy, and the performance was especially remarkable when the input rankings
shared many items, and (2) the two stabilization techniques successfully
improved the evaluation accuracy and efficiency.
