  In this paper, we address the problem of cost-sensitive multi-fidelity
Bayesian Optimization (BO) for efficient hyperparameter optimization (HPO).
Specifically, we assume a scenario where users want to early-stop the BO when
the performance improvement is not satisfactory with respect to the required
computational cost. Motivated by this scenario, we introduce utility, which is
a function predefined by each user and describes the trade-off between cost and
performance of BO. This utility function, combined with our novel acquisition
function and stopping criterion, allows us to dynamically choose for each BO
step the best configuration that we expect to maximally improve the utility in
future, and also automatically stop the BO around the maximum utility. Further,
we improve the sample efficiency of existing learning curve (LC) extrapolation
methods with transfer learning, while successfully capturing the correlations
between different configurations to develop a sensible surrogate function for
multi-fidelity BO. We validate our algorithm on various LC datasets and found
it outperform all the previous multi-fidelity BO and transfer-BO baselines we
consider, achieving significantly better trade-off between cost and performance
of BO.
