  Off-policy evaluation is a key component of reinforcement learning which
evaluates a target policy with offline data collected from behavior policies.
It is a crucial step towards safe reinforcement learning and has been used in
advertisement, recommender systems and many other applications. In these
applications, sometimes the offline data is collected from multiple behavior
policies. Previous works regard data from different behavior policies equally.
Nevertheless, some behavior policies are better at producing good estimators
while others are not. This paper starts with discussing how to correctly mix
estimators produced by different behavior policies. We propose three ways to
reduce the variance of the mixture estimator when all sub-estimators are
unbiased or asymptotically unbiased. Furthermore, experiments on simulated
recommender systems show that our methods are effective in reducing the
Mean-Square Error of estimation.
