  Surrogate algorithms such as Bayesian optimisation are especially designed
for black-box optimisation problems with expensive objectives, such as
hyperparameter tuning or simulation-based optimisation. In the literature,
these algorithms are usually evaluated with synthetic benchmarks which are well
established but have no expensive objective, and only on one or two real-life
applications which vary wildly between papers. There is a clear lack of
standardisation when it comes to benchmarking surrogate algorithms on
real-life, expensive, black-box objective functions. This makes it very
difficult to draw conclusions on the effect of algorithmic contributions and to
give substantial advice on which method to use when. A new benchmark library,
EXPObench, provides first steps towards such a standardisation. The library is
used to provide an extensive comparison of six different surrogate algorithms
on four expensive optimisation problems from different real-life applications.
This has led to new insights regarding the relative importance of exploration,
the evaluation time of the objective, and the used model. We also provide rules
of thumb for which surrogate algorithm to use in which situation. A further
contribution is that we make the algorithms and benchmark problem instances
publicly available, contributing to more uniform analysis of surrogate
algorithms. Most importantly, we include the performance of the six algorithms
on all evaluated problem instances. This results in a unique new dataset that
lowers the bar for researching new methods as the number of expensive
evaluations required for comparison is significantly reduced.
