  This paper introduces a new empirical methodology, the Cross-environment
Hyperparameter Setting Benchmark, that compares RL algorithms across
environments using a single hyperparameter setting, encouraging algorithmic
development which is insensitive to hyperparameters. We demonstrate that this
benchmark is robust to statistical noise and obtains qualitatively similar
results across repeated applications, even when using few samples. This
robustness makes the benchmark computationally cheap to apply, allowing
statistically sound insights at low cost. We demonstrate two example
instantiations of the CHS, on a set of six small control environments (SC-CHS)
and on the entire DM Control suite of 28 environments (DMC-CHS). Finally, to
illustrate the applicability of the CHS to modern RL algorithms on challenging
environments, we conduct a novel empirical study of an open question in the
continuous control literature. We show, with high confidence, that there is no
meaningful difference in performance between Ornstein-Uhlenbeck noise and
uncorrelated Gaussian noise for exploration with the DDPG algorithm on the
DMC-CHS.
