  Open-domain dialog systems have a user-centric goal: to provide humans with
an engaging conversation experience. User engagement is one of the most
important metrics for evaluating open-domain dialog systems, and could also be
used as real-time feedback to benefit dialog policy learning. Existing work on
detecting user disengagement typically requires hand-labeling many dialog
samples. We propose HERALD, an efficient annotation framework that reframes the
training data annotation process as a denoising problem. Specifically, instead
of manually labeling training samples, we first use a set of labeling
heuristics to label training samples automatically. We then denoise the weakly
labeled data using the Shapley algorithm. Finally, we use the denoised data to
train a user engagement detector. Our experiments show that HERALD improves
annotation efficiency significantly and achieves 86% user disengagement
detection accuracy in two dialog corpora.
