  We study preferential Bayesian optimization (BO) where reliable feedback is
limited to pairwise comparison called duels. An important challenge in
preferential BO, which uses the preferential Gaussian process (GP) model to
represent flexible preference structure, is that the posterior distribution is
a computationally intractable skew GP. The most widely used approach for
preferential BO is Gaussian approximation, which ignores the skewness of the
true posterior. Alternatively, Markov chain Monte Carlo (MCMC) based
preferential BO is also proposed. In this work, we first verify the accuracy of
Gaussian approximation, from which we reveal the critical problem that the
predictive probability of duels can be inaccurate. This observation motivates
us to improve the MCMC-based estimation for skew GP, for which we show the
practical efficiency of Gibbs sampling and derive the low variance MC
estimator. However, the computational time of MCMC can still be a bottleneck in
practice. Towards building a more practical preferential BO, we develop a new
method that achieves both high computational efficiency and low sample
complexity, and then demonstrate its effectiveness through extensive numerical
experiments.
