  Off-policy Evaluation (OPE) methods are crucial tools for evaluating policies
in high-stakes domains such as healthcare, where direct deployment is often
infeasible, unethical, or expensive. When deployment environments are expected
to undergo changes (that is, dataset shifts), it is important for OPE methods
to perform robust evaluation of the policies amidst such changes. Existing
approaches consider robustness against a large class of shifts that can
arbitrarily change any observable property of the environment. This often
results in highly pessimistic estimates of the utilities, thereby invalidating
policies that might have been useful in deployment. In this work, we address
the aforementioned problem by investigating how domain knowledge can help
provide more realistic estimates of the utilities of policies. We leverage
human inputs on which aspects of the environments may plausibly change, and
adapt the OPE methods to only consider shifts on these aspects. Specifically,
we propose a novel framework, Robust OPE (ROPE), which considers shifts on a
subset of covariates in the data based on user inputs, and estimates worst-case
utility under these shifts. We then develop computationally efficient
algorithms for OPE that are robust to the aforementioned shifts for contextual
bandits and Markov decision processes. We also theoretically analyze the sample
complexity of these algorithms. Extensive experimentation with synthetic and
real world datasets from the healthcare domain demonstrates that our approach
not only captures realistic dataset shifts accurately, but also results in less
pessimistic policy evaluations.
