  Bayesian optimization (BO) has emerged during the last few years as an
effective approach to optimizing black-box functions where direct queries of
the objective are expensive. In this paper we consider the case where direct
access to the function is not possible, but information about user preferences
is. Such scenarios arise in problems where human preferences are modeled, such
as A/B tests or recommender systems. We present a new framework for this
scenario that we call Preferential Bayesian Optimization (PBO) which allows us
to find the optimum of a latent function that can only be queried through
pairwise comparisons, the so-called duels. PBO extends the applicability of
standard BO ideas and generalizes previous discrete dueling approaches by
modeling the probability of the winner of each duel by means of a Gaussian
process model with a Bernoulli likelihood. The latent preference function is
used to define a family of acquisition functions that extend usual policies
used in BO. We illustrate the benefits of PBO in a variety of experiments,
showing that PBO needs drastically fewer comparisons for finding the optimum.
According to our experiments, the way of modeling correlations in PBO is key in
obtaining this advantage.
