  The stochastic nature of iterative optimization heuristics leads to
inherently noisy performance measurements. Since these measurements are often
gathered once and then used repeatedly, the number of collected samples will
have a significant impact on the reliability of algorithm comparisons. We show
that care should be taken when making decisions based on limited data.
Particularly, we show that the number of runs used in many benchmarking
studies, e.g., the default value of 15 suggested by the COCO environment, can
be insufficient to reliably rank algorithms on well-known numerical
optimization benchmarks.
  Additionally, methods for automated algorithm configuration are sensitive to
insufficient sample sizes. This may result in the configurator choosing a
`lucky' but poor-performing configuration despite exploring better ones. We
show that relying on mean performance values, as many configurators do, can
require a large number of runs to provide accurate comparisons between the
considered configurations. Common statistical tests can greatly improve the
situation in most cases but not always. We show examples of performance losses
of more than 20%, even when using statistical races to dynamically adjust the
number of runs, as done by irace. Our results underline the importance of
appropriately considering the statistical distribution of performance values.
