  Recent progress on neural approaches for language processing has triggered a
resurgence of interest on building intelligent open-domain chatbots. However,
even the state-of-the-art neural chatbots cannot produce satisfying responses
for every turn in a dialog. A practical solution is to generate multiple
response candidates for the same context, and then perform response
ranking/selection to determine which candidate is the best. Previous work in
response selection typically trains response rankers using synthetic data that
is formed from existing dialogs by using a ground truth response as the single
appropriate response and constructing inappropriate responses via random
selection or using adversarial methods. In this work, we curated a dataset
where responses from multiple response generators produced for the same dialog
context are manually annotated as appropriate (positive) and inappropriate
(negative). We argue that such training data better matches the actual use case
examples, enabling the models to learn to rank responses effectively. With this
new dataset, we conduct a systematic evaluation of state-of-the-art methods for
response selection, and demonstrate that both strategies of using multiple
positive candidates and using manually verified hard negative candidates can
bring in significant performance improvement in comparison to using the
adversarial training data, e.g., increase of 3% and 13% in Recall@1 score,
respectively.
