  Multi-armed bandit algorithms like Thompson Sampling (TS) can be used to
conduct adaptive experiments, in which maximizing reward means that data is
used to progressively assign participants to more effective arms. Such
assignment strategies increase the risk of statistical hypothesis tests
identifying a difference between arms when there is not one, and failing to
conclude there is a difference in arms when there truly is one. We tackle this
by introducing a novel heuristic algorithm, called TS-PostDiff (Posterior
Probability of Difference). TS-PostDiff takes a Bayesian approach to mixing TS
and Uniform Random (UR): the probability a participant is assigned using UR
allocation is the posterior probability that the difference between two arms is
'small' (below a certain threshold), allowing for more UR exploration when
there is little or no reward to be gained. We evaluate TS-PostDiff against
state-of-the-art strategies. The empirical and simulation results help
characterize the trade-offs of these approaches between reward, False Positive
Rate (FPR), and statistical power, as well as under which circumstances each is
effective. We quantify the advantage of TS-PostDiff in performing well across
multiple differences in arm means (effect sizes), showing the benefits of
adaptively changing randomization/exploration in TS in a "Statistically
Considerate" manner: reducing FPR and increasing statistical power when
differences are small or zero and there is less reward to be gained, while
exploiting more when differences may be large. This highlights important
considerations for future algorithm development and analysis to better balance
reward and statistical analysis.
