  Many real-world experimental design problems (a) evaluate multiple
experimental conditions in parallel and (b) replicate each condition multiple
times due to large and heteroscedastic observation noise. Given a fixed total
budget, this naturally induces a trade-off between evaluating more unique
conditions while replicating each of them fewer times vs. evaluating fewer
unique conditions and replicating each more times. Moreover, in these problems,
practitioners may be risk-averse and hence prefer an input with both good
average performance and small variability. To tackle both challenges, we
propose the Batch Thompson Sampling for Replicable Experimental Design
(BTS-RED) framework, which encompasses three algorithms. Our BTS-RED-Known and
BTS-RED-Unknown algorithms, for, respectively, known and unknown noise
variance, choose the number of replications adaptively rather than
deterministically such that an input with a larger noise variance is replicated
more times. As a result, despite the noise heteroscedasticity, both algorithms
enjoy a theoretical guarantee and are asymptotically no-regret. Our
Mean-Var-BTS-RED algorithm aims at risk-averse optimization and is also
asymptotically no-regret. We also show the effectiveness of our algorithms in
two practical real-world applications: precision agriculture and AutoML.
