  The goal of off-policy evaluation (OPE) is to evaluate a new policy using
historical data obtained via a behavior policy. However, because the contextual
bandit algorithm updates the policy based on past observations, the samples are
not independent and identically distributed (i.i.d.). This paper tackles this
problem by constructing an estimator from a martingale difference sequence
(MDS) for the dependent samples. In the data-generating process, we do not
assume the convergence of the policy, but the policy uses the same conditional
probability of choosing an action during a certain period. Then, we derive an
asymptotically normal estimator of the value of an evaluation policy. As
another advantage of our method, the batch-based approach simultaneously solves
the deficient support problem. Using benchmark and real-world datasets, we
experimentally confirm the effectiveness of the proposed method.
