  Finding the best configuration of algorithms' hyperparameters for a given
optimization problem is an important task in evolutionary computation. We
compare in this work the results of four different hyperparameter tuning
approaches for a family of genetic algorithms on 25 diverse pseudo-Boolean
optimization problems. More precisely, we compare previously obtained results
from a grid search with those obtained from three automated configuration
techniques: iterated racing, mixed-integer parallel efficient global
optimization, and mixed-integer evolutionary strategies.
  Using two different cost metrics, expected running time and the area under
the empirical cumulative distribution function curve, we find that in several
cases the best configurations with respect to expected running time are
obtained when using the area under the empirical cumulative distribution
function curve as the cost metric during the configuration process. Our results
suggest that even when interested in expected running time performance, it
might be preferable to use anytime performance measures for the configuration
task. We also observe that tuning for expected running time is much more
sensitive with respect to the budget that is allocated to the target
algorithms.
