  Testing whether a variable of interest affects the outcome is one of the most
fundamental problem in statistics and is often the main scientific question of
interest. To tackle this problem, the conditional randomization test (CRT) is
widely used to test the independence of variable(s) of interest (X) with an
outcome (Y) holding other variable(s) (Z) fixed. The CRT uses randomization or
design-based inference that relies solely on the iid sampling of (X,Z) to
produce exact finite-sample p-values that are constructed using any test
statistic. We propose a new method, the adaptive randomization test (ART), that
tackles the independence problem while allowing the data to be adaptively
sampled. We first showcase the ART in a particular multi-arm bandit problem
known as the normal-mean model. Under this setting, we theoretically
characterize the powers of both the iid sampling procedure and the adaptive
sampling procedure and empirically find that the ART can uniformly outperform
the CRT that pulls all arms independently with equal probability. We also
surprisingly find that the ART can be more powerful than even the CRT that uses
an oracle iid sampling procedure when the signal is relatively strong. We
believe that the proposed adaptive procedure is successful because it takes
arms that may initially look like "fake" signals due to random chance and
stabilizes them closer to "null" signals. We additionally showcase the ART to a
popular factorial survey design setting known as conjoint analysis. We find
similar results through simulations and a recent application concerning the
role of gender discrimination in political candidate evaluation.
