  The benchmark datasets for neural architecture search (NAS) have been
developed to alleviate the computationally expensive evaluation process and
ensure a fair comparison. Recent NAS benchmarks only focus on architecture
optimization, although the training hyperparameters affect the obtained model
performances. Building the benchmark dataset for joint optimization of
architecture and training hyperparameters is essential to further NAS research.
The existing NAS-HPO-Bench is a benchmark for joint optimization, but it does
not consider the network connectivity design as done in modern NAS algorithms.
This paper introduces the first benchmark dataset for joint optimization of
network connections and training hyperparameters, which we call
NAS-HPO-Bench-II. We collect the performance data of 4K cell-based
convolutional neural network architectures trained on the CIFAR-10 dataset with
different learning rate and batch size settings, resulting in the data of 192K
configurations. The dataset includes the exact data for 12 epoch training. We
further build the surrogate model predicting the accuracies after 200 epoch
training to provide the performance data of longer training epoch. By analyzing
NAS-HPO-Bench-II, we confirm the dependency between architecture and training
hyperparameters and the necessity of joint optimization. Finally, we
demonstrate the benchmarking of the baseline optimization algorithms using
NAS-HPO-Bench-II.
