  Zero-shot hyperparameter optimization (HPO) is a simple yet effective use of
transfer learning for constructing a small list of hyperparameter (HP)
configurations that complement each other. That is to say, for any given
dataset, at least one of them is expected to perform well. Current techniques
for obtaining this list are computationally expensive as they rely on running
training jobs on a diverse collection of datasets and a large collection of
randomly drawn HPs. This cost is especially problematic in environments where
the space of HPs is regularly changing due to new algorithm versions, or
changing architectures of deep networks. We provide an overview of available
approaches and introduce two novel techniques to handle the problem. The first
is based on a surrogate model and adaptively chooses pairs of dataset,
configuration to query. The second, for settings where finding, tuning and
testing a surrogate model is problematic, is a multi-fidelity technique
combining HyperBand with submodular optimization. We benchmark our methods
experimentally on five tasks (XGBoost, LightGBM, CatBoost, MLP and AutoML) and
show significant improvement in accuracy compared to standard zero-shot HPO
with the same training budget. In addition to contributing new algorithms, we
provide an extensive study of the zero-shot HPO technique resulting in (1)
default hyper-parameters for popular algorithms that would benefit the
community using them, (2) massive lookup tables to further the research of
hyper-parameter tuning.
