  Hyperparameter optimization (HPO) is a key component of machine learning
models for achieving peak predictive performance. While numerous methods and
algorithms for HPO have been proposed over the last years, little progress has
been made in illuminating and examining the actual structure of these black-box
optimization problems. Exploratory landscape analysis (ELA) subsumes a set of
techniques that can be used to gain knowledge about properties of unknown
optimization problems. In this paper, we evaluate the performance of five
different black-box optimizers on 30 HPO problems, which consist of two-,
three- and five-dimensional continuous search spaces of the XGBoost learner
trained on 10 different data sets. This is contrasted with the performance of
the same optimizers evaluated on 360 problem instances from the black-box
optimization benchmark (BBOB). We then compute ELA features on the HPO and BBOB
problems and examine similarities and differences. A cluster analysis of the
HPO and BBOB problems in ELA feature space allows us to identify how the HPO
problems compare to the BBOB problems on a structural meta-level. We identify a
subset of BBOB problems that are close to the HPO problems in ELA feature space
and show that optimizer performance is comparably similar on these two sets of
benchmark problems. We highlight open challenges of ELA for HPO and discuss
potential directions of future research and applications.
