  Multi armed bandit (MAB) algorithms have been increasingly used to complement
or integrate with A/B tests and randomized clinical trials in e-commerce,
healthcare, and policymaking. Recent developments incorporate possible delayed
feedback. While existing MAB literature often focuses on maximizing the
expected cumulative reward outcomes (or, equivalently, regret minimization),
few efforts have been devoted to establish valid statistical inference
approaches to quantify the uncertainty of learned policies. We attempt to fill
this gap by providing a unified statistical inference framework for policy
evaluation where a target policy is allowed to differ from the data collecting
policy, and our framework allows delay to be associated with the treatment
arms. We present an adaptively weighted estimator that on one hand incorporates
the arm-dependent delaying mechanism to achieve consistency, and on the other
hand mitigates the variance inflation across stages due to vanishing sampling
probability. In particular, our estimator does not critically depend on the
ability to estimate the unknown delay mechanism. Under appropriate conditions,
we prove that our estimator converges to a normal distribution as the number of
time points goes to infinity, which provides guarantees for large-sample
statistical inference. We illustrate the finite-sample performance of our
approach through Monte Carlo experiments.
