  Hyperparameter optimization (HPO) is a well-studied research field. However,
the effects and interactions of the components in an HPO pipeline are not yet
well investigated. Then, we ask ourselves: can the landscape of HPO be biased
by the pipeline used to evaluate individual configurations? To address this
question, we proposed to analyze the effect of the HPO pipeline on HPO problems
using fitness landscape analysis. Particularly, we studied the DS-2019 HPO
benchmark data set, looking for patterns that could indicate evaluation
pipeline malfunction, and relate them to HPO performance. Our main findings
are: (i) In most instances, large groups of diverse hyperparameters (i.e.,
multiple configurations) yield the same ill performance, most likely associated
with majority class prediction models; (ii) in these cases, a worsened
correlation between the observed fitness and average fitness in the
neighborhood is observed, potentially making harder the deployment of
local-search based HPO strategies. Finally, we concluded that the HPO pipeline
definition might negatively affect the HPO landscape.
