  In bandit algorithms, the randomly time-varying adaptive experimental design
makes it difficult to apply traditional limit theorems to off-policy evaluation
of the treatment effect. Moreover, the normal approximation by the central
limit theorem becomes unsatisfactory for lack of information due to the small
sample size of the inferior arm. To resolve this issue, we introduce a
backwards asymptotic expansion method and prove the validity of this scheme
based on the partial mixing, that was originally introduced for the expansion
of the distribution of a functional of a jump-diffusion process in a random
environment. The theory is generalized in this paper to incorporate the
backward propagation of random functions in the bandit algorithm. Besides the
analytical validation, the simulation studies also support the new method. Our
formulation is general and applicable to nonlinearly parametrized
differentiable statistical models having an adaptive design.
