  Feature-based algorithm selection aims to automatically find the best one
from a portfolio of optimization algorithms on an unseen problem based on its
landscape features. Feature-based algorithm selection has recently received
attention in the research field of black-box numerical optimization. However,
there is still room for analysis of algorithm selection for black-box
optimization. Most previous studies have focused only on whether an algorithm
selection system can outperform the single-best solver in a portfolio. In
addition, a benchmarking methodology for algorithm selection systems has not
been well investigated in the literature. In this context, this paper analyzes
algorithm selection systems on the 24 noiseless black-box optimization
benchmarking functions. First, we demonstrate that the first successful
performance measure is more reliable than the expected runtime measure for
benchmarking algorithm selection systems. Then, we examine the influence of
randomness on the performance of algorithm selection systems. We also show that
the performance of algorithm selection systems can be significantly improved by
using sequential least squares programming as a pre-solver. We point out that
the difficulty of outperforming the single-best solver depends on algorithm
portfolios, cross-validation methods, and dimensions. Finally, we demonstrate
that the effectiveness of algorithm portfolios depends on various factors.
These findings provide fundamental insights for algorithm selection for
black-box optimization.
