  Contextual bandit algorithms provide principled online learning solutions to
balance the exploitation-exploration trade-off in various applications such as
recommender systems. However, the learning speed of the traditional contextual
bandit algorithms is often slow due to the need for extensive exploration. This
poses a critical issue in applications like recommender systems, since users
may need to provide feedbacks on a lot of uninterested items. To accelerate the
learning speed, we generalize contextual bandit to conversational contextual
bandit. Conversational contextual bandit leverages not only behavioral
feedbacks on arms (e.g., articles in news recommendation), but also occasional
conversational feedbacks on key-terms from the user. Here, a key-term can
relate to a subset of arms, for example, a category of articles in news
recommendation. We then design the Conversational UCB algorithm (ConUCB) to
address two challenges in conversational contextual bandit: (1) which key-terms
to select to conduct conversation, (2) how to leverage conversational feedbacks
to accelerate the speed of bandit learning. We theoretically prove that ConUCB
can achieve a smaller regret upper bound than the traditional contextual bandit
algorithm LinUCB, which implies a faster learning speed. Experiments on
synthetic data, as well as real datasets from Yelp and Toutiao, demonstrate the
efficacy of the ConUCB algorithm.
