  The performance of many machine learning algorithms depends on their
hyperparameter settings. The goal of this study is to determine whether it is
important to tune a hyperparameter or whether it can be safely set to a default
value. We present a methodology to determine the importance of tuning a
hyperparameter based on a non-inferiority test and tuning risk: the performance
loss that is incurred when a hyperparameter is not tuned, but set to a default
value. Because our methods require the notion of a default parameter, we
present a simple procedure that can be used to determine reasonable default
parameters. We apply our methods in a benchmark study using 59 datasets from
OpenML. Our results show that leaving particular hyperparameters at their
default value is non-inferior to tuning these hyperparameters. In some cases,
leaving the hyperparameter at its default value even outperforms tuning it
using a search procedure with a limited number of iterations.
