  In many real-world applications, we are interested in approximating
black-box, costly functions as accurately as possible with the smallest number
of function evaluations. A complex computer code is an example of such a
function. In this work, a Gaussian process (GP) emulator is used to approximate
the output of complex computer code. We consider the problem of extending an
initial experiment (set of model runs) sequentially to improve the emulator. A
sequential sampling approach based on leave-one-out (LOO) cross-validation is
proposed that can be easily extended to a batch mode. This is a desirable
property since it saves the user time when parallel computing is available.
After fitting a GP to training data points, the expected squared LOO (ES-LOO)
error is calculated at each design point. ES-LOO is used as a measure to
identify important data points. More precisely, when this quantity is large at
a point it means that the quality of prediction depends a great deal on that
point and adding more samples nearby could improve the accuracy of the GP. As a
result, it is reasonable to select the next sample where ES-LOO is maximised.
However, ES-LOO is only known at the experimental design and needs to be
estimated at unobserved points. To do this, a second GP is fitted to the ES-LOO
errors and where the maximum of the modified expected improvement (EI)
criterion occurs is chosen as the next sample. EI is a popular acquisition
function in Bayesian optimisation and is used to trade-off between local/global
search. However, it has a tendency towards exploitation, meaning that its
maximum is close to the (current) "best" sample. To avoid clustering, a
modified version of EI, called pseudo expected improvement, is employed which
is more explorative than EI yet allows us to discover unexplored regions. Our
results show that the proposed sampling method is promising.
