  Many optimizers have been proposed for training deep neural networks, and
they often have multiple hyperparameters, which make it tricky to benchmark
their performance. In this work, we propose a new benchmarking protocol to
evaluate both end-to-end efficiency (training a model from scratch without
knowing the best hyperparameter) and data-addition training efficiency (the
previously selected hyperparameters are used for periodically re-training the
model with newly collected data). For end-to-end efficiency, unlike previous
work that assumes random hyperparameter tuning, which over-emphasizes the
tuning time, we propose to evaluate with a bandit hyperparameter tuning
strategy. A human study is conducted to show that our evaluation protocol
matches human tuning behavior better than the random search. For data-addition
training, we propose a new protocol for assessing the hyperparameter
sensitivity to data shift. We then apply the proposed benchmarking framework to
7 optimizers and various tasks, including computer vision, natural language
processing, reinforcement learning, and graph mining. Our results show that
there is no clear winner across all the tasks.
