  Contextual bandit algorithms are ubiquitous tools for active sequential
experimentation in healthcare and the tech industry. They involve online
learning algorithms that adaptively learn policies over time to map observed
contexts $X_t$ to actions $A_t$ in an attempt to maximize stochastic rewards
$R_t$. This adaptivity raises interesting but hard statistical inference
questions, especially counterfactual ones: for example, it is often of interest
to estimate the properties of a hypothetical policy that is different from the
logging policy that was used to collect the data -- a problem known as
``off-policy evaluation'' (OPE). Using modern martingale techniques, we present
a comprehensive framework for OPE inference that relax many unnecessary
assumptions made in past work, significantly improving on them both
theoretically and empirically. Importantly, our methods can be employed while
the original experiment is still running (that is, not necessarily post-hoc),
when the logging policy may be itself changing (due to learning), and even if
the context distributions are a highly dependent time-series (such as if they
are drifting over time). More concretely, we derive confidence sequences for
various functionals of interest in OPE. These include doubly robust ones for
time-varying off-policy mean reward values, but also confidence bands for the
entire CDF of the off-policy reward distribution. All of our methods (a) are
valid at arbitrary stopping times (b) only make nonparametric assumptions, (c)
do not require known bounds on the maximal importance weights, and (d) adapt to
the empirical variance of our estimators. In summary, our methods enable
anytime-valid off-policy inference using adaptively collected contextual bandit
data.
