  We run an independent comparison of all hyperparameter optimization
(hyperopt) engines available in the Ray Tune library. We introduce two ways to
normalize and aggregate statistics across data sets and models, one rank-based,
and another one sandwiching the score between the random search score and the
full grid search score. This affords us i) to rank the hyperopt engines, ii) to
make generalized and statistically significant statements on how much they
improve over random search, and iii) to make recommendations on which engine
should be used to hyperopt a given learning algorithm. We find that most
engines beat random search, but that only three of them (HEBO, AX, and
BlendSearch) clearly stand out. We also found that some engines seem to
specialize in hyperopting certain learning algorithms, which makes it tricky to
use hyperopt in comparison studies, since the choice of the hyperopt technique
may favor some of the models in the comparison.
