  For many NLP applications, such as question answering and summarisation, the
goal is to select the best solution from a large space of candidates to meet a
particular user's needs. To address the lack of user-specific training data, we
propose an interactive text ranking approach that actively selects pairs of
candidates, from which the user selects the best. Unlike previous strategies,
which attempt to learn a ranking across the whole candidate space, our method
employs Bayesian optimisation to focus the user's labelling effort on high
quality candidates and integrates prior knowledge in a Bayesian manner to cope
better with small data scenarios. We apply our method to community question
answering (cQA) and extractive summarisation, finding that it significantly
outperforms existing interactive approaches. We also show that the ranking
function learned by our method is an effective reward function for
reinforcement learning, which improves the state of the art for interactive
summarisation.
