  We consider a model-based approach to perform batch off-policy evaluation in
reinforcement learning. Our method takes a mixture-of-experts approach to
combine parametric and non-parametric models of the environment such that the
final value estimate has the least expected error. We do so by first estimating
the local accuracy of each model and then using a planner to select which model
to use at every time step as to minimize the return error estimate along entire
trajectories. Across a variety of domains, our mixture-based approach
outperforms the individual models alone as well as state-of-the-art importance
sampling-based estimators.
