  We show that the popular reinforcement learning (RL) strategy of estimating
the state-action value (Q-function) by minimizing the mean squared Bellman
error leads to a regression problem with confounding, the inputs and output
noise being correlated. Hence, direct minimization of the Bellman error can
result in significantly biased Q-function estimates. We explain why fixing the
target Q-network in Deep Q-Networks and Fitted Q Evaluation provides a way of
overcoming this confounding, thus shedding new light on this popular but not
well understood trick in the deep RL literature. An alternative approach to
address confounding is to leverage techniques developed in the causality
literature, notably instrumental variables (IV). We bring together here the
literature on IV and RL by investigating whether IV approaches can lead to
improved Q-function estimates. This paper analyzes and compares a wide range of
recent IV methods in the context of offline policy evaluation (OPE), where the
goal is to estimate the value of a policy using logged data only. By applying
different IV techniques to OPE, we are not only able to recover previously
proposed OPE methods such as model-based techniques but also to obtain
competitive new techniques. We find empirically that state-of-the-art OPE
methods are closely matched in performance by some IV methods such as AGMM,
which were not developed for OPE. We open-source all our code and datasets at
https://github.com/liyuan9988/IVOPEwithACME.
