  We study the problem of off-policy evaluation (OPE) for episodic Partially
Observable Markov Decision Processes (POMDPs) with continuous states. Motivated
by the recently proposed proximal causal inference framework, we develop a
non-parametric identification result for estimating the policy value via a
sequence of so-called V-bridge functions with the help of time-dependent proxy
variables. We then develop a fitted-Q-evaluation-type algorithm to estimate
V-bridge functions recursively, where a non-parametric instrumental variable
(NPIV) problem is solved at each step. By analyzing this challenging sequential
NPIV problem, we establish the finite-sample error bounds for estimating the
V-bridge functions and accordingly that for evaluating the policy value, in
terms of the sample size, length of horizon and so-called (local) measure of
ill-posedness at each step. To the best of our knowledge, this is the first
finite-sample error bound for OPE in POMDPs under non-parametric models.
