  Hyperparameter plays an essential role in the fitting of supervised machine
learning algorithms. However, it is computationally expensive to tune all the
tunable hyperparameters simultaneously especially for large data sets. In this
paper, we give a definition of hyperparameter importance that can be estimated
by subsampling procedures. According to the importance, hyperparameters can
then be tuned on the entire data set more efficiently. We show theoretically
that the proposed importance on subsets of data is consistent with the one on
the population data under weak conditions. Numerical experiments show that the
proposed importance is consistent and can save a lot of computational
resources.
