  Sequential maximization of expected improvement (EI) is one of the most
widely used policies in Bayesian optimization because of its simplicity and
ability to handle noisy observations. In particular, the improvement function
often uses the best posterior mean as the best incumbent in noisy settings.
However, the uncertainty associated with the incumbent solution is often
neglected in many analytic EI-type methods: a closed-form acquisition function
is derived in the noise-free setting, but then applied to the setting with
noisy observations. To address this limitation, we propose a modification of EI
that corrects its closed-form expression by incorporating the covariance
information provided by the Gaussian Process (GP) model. This acquisition
function specializes to the classical noise-free result, and we argue should
replace that formula in Bayesian optimization software packages, tutorials, and
textbooks. This enhanced acquisition provides good generality for noisy and
noiseless settings. We show that our method achieves a sublinear convergence
rate on the cumulative regret bound under heteroscedastic observation noise.
Our empirical results demonstrate that our proposed acquisition function can
outperform EI in the presence of noisy observations on benchmark functions for
black-box optimization, as well as on parameter search for neural network model
compression.
