  Many state-of-the-art automated machine learning (AutoML) systems use greedy
ensemble selection (GES) by Caruana et al. (2004) to ensemble models found
during model selection post hoc. Thereby, boosting predictive performance and
likely following Auto-Sklearn 1's insight that alternatives, like stacking or
gradient-free numerical optimization, overfit. Overfitting in Auto-Sklearn 1 is
much more likely than in other AutoML systems because it uses only low-quality
validation data for post hoc ensembling. Therefore, we were motivated to
analyze whether Auto-Sklearn 1's insight holds true for systems with
higher-quality validation data. Consequently, we compared the performance of
covariance matrix adaptation evolution strategy (CMA-ES), state-of-the-art
gradient-free numerical optimization, to GES on the 71 classification datasets
from the AutoML benchmark for AutoGluon. We found that Auto-Sklearn's insight
depends on the chosen metric. For the metric ROC AUC, CMA-ES overfits
drastically and is outperformed by GES -- statistically significantly for
multi-class classification. For the metric balanced accuracy, CMA-ES does not
overfit and outperforms GES significantly. Motivated by the successful
application of CMA-ES for balanced accuracy, we explored methods to stop CMA-ES
from overfitting for ROC AUC. We propose a method to normalize the weights
produced by CMA-ES, inspired by GES, that avoids overfitting for CMA-ES and
makes CMA-ES perform better than or similar to GES for ROC AUC.
