  This paper addresses the problem of policy selection in domains with abundant
logged data, but with a restricted interaction budget. Solving this problem
would enable safe evaluation and deployment of offline reinforcement learning
policies in industry, robotics, and recommendation domains among others.
Several off-policy evaluation (OPE) techniques have been proposed to assess the
value of policies using only logged data. However, there is still a big gap
between the evaluation by OPE and the full online evaluation. Yet, large
amounts of online interactions are often not possible in practice. To overcome
this problem, we introduce active offline policy selection - a novel sequential
decision approach that combines logged data with online interaction to identify
the best policy. We use OPE estimates to warm start the online evaluation.
Then, in order to utilize the limited environment interactions wisely we decide
which policy to evaluate next based on a Bayesian optimization method with a
kernel that represents policy similarity. We use multiple benchmarks, including
real-world robotics, with a large number of candidate policies to show that the
proposed approach improves upon state-of-the-art OPE estimates and pure online
policy evaluation.
