  Previous efforts on hyperparameter optimization (HPO) of machine learning
(ML) models predominately focus on algorithmic advances, yet little is known
about the topography of the underlying hyperparameter (HP) loss landscape,
which plays a fundamental role in governing the search process of HPO. While
several works have conducted fitness landscape analysis (FLA) on various ML
systems, they are limited to properties of isolated landscape without
interrogating the potential structural similarities among them. The exploration
of such similarities can provide a novel perspective for understanding the
mechanism behind modern HPO methods, but has been missing, possibly due to the
expensive cost of large-scale landscape construction, and the lack of effective
analysis methods. In this paper, we mapped 1,500 HP loss landscapes of 6
representative ML models on 63 datasets across different fidelity levels, with
11M+ configurations. By conducting exploratory analysis on these landscapes
with fine-grained visualizations and dedicated FLA metrics, we observed a
similar landscape topography across a wide range of models, datasets, and
fidelities, and shed light on several central topics in HPO.
