  Off-policy evaluation in reinforcement learning offers the chance of using
observational data to improve future outcomes in domains such as healthcare and
education, but safe deployment in high stakes settings requires ways of
assessing its validity. Traditional measures such as confidence intervals may
be insufficient due to noise, limited data and confounding. In this paper we
develop a method that could serve as a hybrid human-AI system, to enable human
experts to analyze the validity of policy evaluation estimates. This is
accomplished by highlighting observations in the data whose removal will have a
large effect on the OPE estimate, and formulating a set of rules for choosing
which ones to present to domain experts for validation. We develop methods to
compute exactly the influence functions for fitted Q-evaluation with two
different function classes: kernel-based and linear least squares, as well as
importance sampling methods. Experiments on medical simulations and real-world
intensive care unit data demonstrate that our method can be used to identify
limitations in the evaluation process and make evaluation more robust.
