  To achieve peak predictive performance, hyperparameter optimization (HPO) is
a crucial component of machine learning and its applications. Over the last
years, the number of efficient algorithms and tools for HPO grew substantially.
At the same time, the community is still lacking realistic, diverse,
computationally cheap, and standardized benchmarks. This is especially the case
for multi-fidelity HPO methods. To close this gap, we propose HPOBench, which
includes 7 existing and 5 new benchmark families, with a total of more than 100
multi-fidelity benchmark problems. HPOBench allows to run this extendable set
of multi-fidelity HPO benchmarks in a reproducible way by isolating and
packaging the individual benchmarks in containers. It also provides surrogate
and tabular benchmarks for computationally affordable yet statistically sound
evaluations. To demonstrate HPOBench's broad compatibility with various
optimization tools, as well as its usefulness, we conduct an exemplary
large-scale study evaluating 13 optimizers from 6 optimization tools. We
provide HPOBench here: https://github.com/automl/HPOBench.
