  Adaptive experiments use preliminary analyses of the data to inform further
course of action and are commonly used in many disciplines including medical
and social sciences. Because the null hypothesis and experimental design are
not pre-specified, it has long been recognized that statistical inference for
adaptive experiments is not straightforward. Most existing methods only apply
to specific adaptive designs and rely on strong assumptions. In this work, we
propose selective randomization inference as a general framework for analyzing
adaptive experiments. In a nutshell, our approach applies conditional
post-selection inference to randomization tests. By using directed acyclic
graphs to describe the data generating process, we derive a selective
randomization p-value that controls the selective type-I error without
requiring independent and identically distributed data or any other modelling
assumptions. We show how rejection sampling and Markov Chain Monte Carlo can be
used to compute the selective randomization p-values and construct confidence
intervals for a homogeneous treatment effect. To mitigate the risk of
disconnected confidence intervals, we propose the use of hold-out units.
Lastly, we demonstrate our method and compare it with other randomization tests
using synthetic and real-world data.
