  Research on new optimization algorithms is often funded based on the
motivation that such algorithms might improve the capabilities to deal with
real-world and industrially relevant optimization challenges. Besides a huge
variety of different evolutionary and metaheuristic optimization algorithms,
also a large number of test problems and benchmark suites have been developed
and used for comparative assessments of algorithms, in the context of global,
continuous, and black-box optimization. For many of the commonly used synthetic
benchmark problems or artificial fitness landscapes, there are however, no
methods available, to relate the resulting algorithm performance assessments to
technologically relevant real-world optimization problems, or vice versa. Also,
from a theoretical perspective, many of the commonly used benchmark problems
and approaches have little to no generalization value. Based on a mini-review
of publications with critical comments, advice, and new approaches, this
communication aims to give a constructive perspective on several open
challenges and prospective research directions related to systematic and
generalizable benchmarking for black-box optimization.
