  Hyperparameter tuning is one of the the most time-consuming parts in machine
learning. Despite the existence of modern optimization algorithms that minimize
the number of evaluations needed, evaluations of a single setting may still be
expensive. Usually a resampling technique is used, where the machine learning
method has to be fitted a fixed number of k times on different training
datasets. The respective mean performance of the k fits is then used as
performance estimator. Many hyperparameter settings could be discarded after
less than k resampling iterations if they are clearly inferior to
high-performing settings. However, resampling is often performed until the very
end, wasting a lot of computational effort. To this end, we propose the
Sequential Random Search (SQRS) which extends the regular random search
algorithm by a sequential testing procedure aimed at detecting and eliminating
inferior parameter configurations early. We compared our SQRS with regular
random search using multiple publicly available regression and classification
datasets. Our simulation study showed that the SQRS is able to find similarly
well-performing parameter settings while requiring noticeably fewer
evaluations. Our results underscore the potential for integrating sequential
tests into hyperparameter tuning.
