  Automated hyperparameter optimization (HPO) has gained great popularity and
is an important ingredient of most automated machine learning frameworks. The
process of designing HPO algorithms, however, is still an unsystematic and
manual process: Limitations of prior work are identified and the improvements
proposed are -- even though guided by expert knowledge -- still somewhat
arbitrary. This rarely allows for gaining a holistic understanding of which
algorithmic components are driving performance, and carries the risk of
overlooking good algorithmic design choices. We present a principled approach
to automated benchmark-driven algorithm design applied to multifidelity HPO
(MF-HPO): First, we formalize a rich space of MF-HPO candidates that includes,
but is not limited to common HPO algorithms, and then present a configurable
framework covering this space. To find the best candidate automatically and
systematically, we follow a programming-by-optimization approach and search
over the space of algorithm candidates via Bayesian optimization. We challenge
whether the found design choices are necessary or could be replaced by more
naive and simpler ones by performing an ablation analysis. We observe that
using a relatively simple configuration, in some ways simpler than established
methods, performs very well as long as some critical configuration parameters
have the right value.
