  Despite the evolution of Convolutional Neural Networks (CNNs), their
performance is surprisingly dependent on the choice of hyperparameters.
However, it remains challenging to efficiently explore large hyperparameter
search space due to the long training times of modern CNNs. Multi-fidelity
optimization enables the exploration of more hyperparameter configurations
given budget by early termination of unpromising configurations. However, it
often results in selecting a sub-optimal configuration as training with the
high-performing configuration typically converges slowly in an early phase. In
this paper, we propose Multi-fidelity Optimization with a Recurring Learning
rate (MORL) which incorporates CNNs' optimization process into multi-fidelity
optimization. MORL alleviates the problem of slow-starter and achieves a more
precise low-fidelity approximation. Our comprehensive experiments on general
image classification, transfer learning, and semi-supervised learning
demonstrate the effectiveness of MORL over other multi-fidelity optimization
methods such as Successive Halving Algorithm (SHA) and Hyperband. Furthermore,
it achieves significant performance improvements over hand-tuned hyperparameter
configuration within a practical budget.
