  Controller tuning based on black-box optimization allows to automatically
tune performance-critical parameters w.r.t. mostly arbitrary high-level
closed-loop control objectives. However, a comprehensive benchmark of different
black-box optimizers for control engineering problems has not yet been
conducted. Therefore, in this contribution, 11 different versions of Bayesian
optimization (BO) are compared with seven metaheuristics and other baselines on
a set of ten deterministic simulative single-objective tuning problems in
control. Results indicate that deterministic noise, low multimodality, and
substantial areas with infeasible parametrizations (crash constraints)
characterize control engineering tuning problems. Therefore, a flexible method
to handle crash constraints with BO is presented. A resulting increase in
sample efficiency is shown in comparison to standard BO. Furthermore, benchmark
results indicate that pattern search (PS) performs best on a budget of 25 d
objective function evaluations and a problem dimensionality d of d = 2.
Bayesian adaptive direct search, a combination of BO and PS, is shown to be
most sample efficient for 3 <= d <= 5. Using these optimizers instead of random
search increases controller performance by on average 6.6% and up to 16.1%.
