  Innovations across science and industry are evaluated using randomized trials
(a.k.a. A/B tests). While simple and robust, such static designs are
inefficient or infeasible for testing many hypotheses. Adaptive designs can
greatly improve statistical power in theory, but they have seen limited
adoption due to their fragility in practice. We present a benchmark for
adaptive experimentation based on real-world datasets, highlighting prominent
practical challenges to operationalizing adaptivity: non-stationarity,
batched/delayed feedback, multiple outcomes and objectives, and external
validity. Our benchmark aims to spur methodological development that puts
practical performance (e.g., robustness) as a central concern, rather than
mathematical guarantees on contrived instances. We release an open source
library, AExGym, which is designed with modularity and extensibility in mind to
allow experimentation practitioners to develop custom environments and
algorithms.
