  Many open-domain dialogue models pre-trained with social media comments can
generate coherent replies but have difficulties producing engaging responses
when interacting with real users. This phenomenon might mainly result from the
deficiency of annotated human-human conversations and the misalignment with
human preference. In this paper, we propose a novel and efficient approach
Diamante to boost the open-domain chatbot, where two kinds of human feedback
(including explicit demonstration and implicit preference) are collected and
leveraged. By asking annotators to select or amend the model-generated
candidate responses, Diamante efficiently collects the human demonstrated
responses and constructs a Chinese chit-chat dataset. To enhance the alignment
with human preference, Diamante leverages the implicit preference in the data
collection process and introduces the generation-evaluation joint training.
Comprehensive experiments indicate that the Diamante dataset and joint training
paradigm can significantly boost the performance of Chinese pre-trained
dialogue models.
