  Given a Hyperparameter Optimization(HPO) problem, how to design an algorithm
to find optimal configurations efficiently? Bayesian Optimization(BO) and the
multi-fidelity BO methods employ surrogate models to sample configurations
based on history evaluations. More recent studies obtain better performance by
integrating BO with HyperBand(HB), which accelerates evaluation by early
stopping mechanism. However, these methods ignore the advantage of a suitable
evaluation scheme over the default HyperBand, and the capability of BO is still
constrained by skewed evaluation results. In this paper, we propose FlexHB, a
new method pushing multi-fidelity BO to the limit as well as re-designing a
framework for early stopping with Successive Halving(SH). Comprehensive study
on FlexHB shows that (1) our fine-grained fidelity method considerably enhances
the efficiency of searching optimal configurations, (2) our FlexBand framework
(self-adaptive allocation of SH brackets, and global ranking of configurations
in both current and past SH procedures) grants the algorithm with more
flexibility and improves the anytime performance. Our method achieves superior
efficiency and outperforms other methods on various HPO tasks. Empirical
results demonstrate that FlexHB can achieve up to 6.9X and 11.1X speedups over
the state-of-the-art MFES-HB and BOHB respectively.
