  A recently popular approach to solving reinforcement learning is with data
from human preferences. In fact, human preference data are now used with
classic reinforcement learning algorithms such as actor-critic methods, which
involve evaluating an intermediate policy over a reward learned from human
preference data with distribution shift, known as off-policy evaluation (OPE).
Such algorithm includes (i) learning reward function from human preference
dataset, and (ii) learning expected cumulative reward of a target policy.
Despite the huge empirical success, existing OPE methods with preference data
often lack theoretical understanding and rely heavily on heuristics. In this
paper, we study the sample efficiency of OPE with human preference and
establish a statistical guarantee for it. Specifically, we approach OPE by
learning the value function by fitted-Q-evaluation with a deep neural network.
By appropriately selecting the size of a ReLU network, we show that one can
leverage any low-dimensional manifold structure in the Markov decision process
and obtain a sample-efficient estimator without suffering from the curse of
high data ambient dimensionality. Under the assumption of high reward
smoothness, our results \textit{almost align with the classical OPE results
with observable reward data}. To the best of our knowledge, this is the first
result that establishes a \textit{provably efficient} guarantee for off-policy
evaluation with RLHF.
