  After developer adjustments to a machine learning (ML) algorithm, how can the
results of an old hyperparameter optimization (HPO) automatically be used to
speedup a new HPO? This question poses a challenging problem, as developer
adjustments can change which hyperparameter settings perform well, or even the
hyperparameter search space itself. While many approaches exist that leverage
knowledge obtained on previous tasks, so far, knowledge from previous
development steps remains entirely untapped. In this work, we remedy this
situation and propose a new research framework: hyperparameter transfer across
adjustments (HT-AA). To lay a solid foundation for this research framework, we
provide four simple HT-AA baseline algorithms and eight benchmarks changing
various aspects of ML algorithms, their hyperparameter search spaces, and the
neural architectures used. The best baseline, on average and depending on the
budgets for the old and new HPO, reaches a given performance 1.2--2.6x faster
than a prominent HPO algorithm without transfer. As HPO is a crucial step in ML
development but requires extensive computational resources, this speedup would
lead to faster development cycles, lower costs, and reduced environmental
impacts. To make these benefits available to ML developers off-the-shelf and to
facilitate future research on HT-AA, we provide python packages for our
baselines and benchmarks.
