  This paper is concerned with constructing a confidence interval for a target
policy's value offline based on a pre-collected observational data in infinite
horizon settings. Most of the existing works assume no unmeasured variables
exist that confound the observed actions. This assumption, however, is likely
to be violated in real applications such as healthcare and technological
industries. In this paper, we show that with some auxiliary variables that
mediate the effect of actions on the system dynamics, the target policy's value
is identifiable in a confounded Markov decision process. Based on this result,
we develop an efficient off-policy value estimator that is robust to potential
model misspecification and provide rigorous uncertainty quantification. Our
method is justified by theoretical results, simulated and real datasets
obtained from ridesharing companies. A Python implementation of the proposed
procedure is available at https://github.com/Mamba413/cope.
