  Randomization testing is a fundamental method in statistics, enabling
inferential tasks such as testing for (conditional) independence of random
variables, constructing confidence intervals in semiparametric location models,
and constructing (by inverting a permutation test) model-free prediction
intervals via conformal inference. Randomization tests are exactly valid for
any sample size, but their use is generally confined to exchangeable data. Yet
in many applications, data is routinely collected adaptively via, e.g.,
(contextual) bandit and reinforcement learning algorithms or adaptive
experimental designs. In this paper we present a general framework for
randomization testing on adaptively collected data (despite its
non-exchangeability) that uses a novel weighted randomization test, for which
we also present novel computationally tractable resampling algorithms for
various popular adaptive assignment algorithms, data-generating environments,
and types of inferential tasks. Finally, we demonstrate via a range of
simulations the efficacy of our framework for both testing and
confidence/prediction interval construction.
