  In machine learning, the notion of multi-armed bandits refers to a class of
online learning problems, in which an agent is supposed to simultaneously
explore and exploit a given set of choice alternatives in the course of a
sequential decision process. In the standard setting, the agent learns from
stochastic feedback in the form of real-valued rewards. In many applications,
however, numerical reward signals are not readily available -- instead, only
weaker information is provided, in particular relative preferences in the form
of qualitative comparisons between pairs of alternatives. This observation has
motivated the study of variants of the multi-armed bandit problem, in which
more general representations are used both for the type of feedback to learn
from and the target of prediction. The aim of this paper is to provide a survey
of the state of the art in this field, referred to as preference-based
multi-armed bandits or dueling bandits. To this end, we provide an overview of
problems that have been considered in the literature as well as methods for
tackling them. Our taxonomy is mainly based on the assumptions made by these
methods about the data-generating process and, related to this, the properties
of the preference-based feedback.
