  Hyperparameters in machine learning (ML) have received a fair amount of
attention, and hyperparameter tuning has come to be regarded as an important
step in the ML pipeline. But just how useful is said tuning? While
smaller-scale experiments have been previously conducted, herein we carry out a
large-scale investigation, specifically, one involving 26 ML algorithms, 250
datasets (regression and both binary and multinomial classification), 6 score
metrics, and 28,857,600 algorithm runs. Analyzing the results we conclude that
for many ML algorithms we should not expect considerable gains from
hyperparameter tuning on average, however, there may be some datasets for which
default hyperparameters perform poorly, this latter being truer for some
algorithms than others. By defining a single hp_score value, which combines an
algorithm's accumulated statistics, we are able to rank the 26 ML algorithms
from those expected to gain the most from hyperparameter tuning to those
expected to gain the least. We believe such a study may serve ML practitioners
at large.
