  Most studies on abstractive summarization report ROUGE scores between system
and reference summaries. However, we have a concern about the truthfulness of
generated summaries: whether all facts of a generated summary are mentioned in
the source text. This paper explores improving the truthfulness in headline
generation on two popular datasets. Analyzing headlines generated by the
state-of-the-art encoder-decoder model, we show that the model sometimes
generates untruthful headlines. We conjecture that one of the reasons lies in
untruthful supervision data used for training the model. In order to quantify
the truthfulness of article-headline pairs, we consider the textual entailment
of whether an article entails its headline. After confirming quite a few
untruthful instances in the datasets, this study hypothesizes that removing
untruthful instances from the supervision data may remedy the problem of the
untruthful behaviors of the model. Building a binary classifier that predicts
an entailment relation between an article and its headline, we filter out
untruthful instances from the supervision data. Experimental results
demonstrate that the headline generation model trained on filtered supervision
data shows no clear difference in ROUGE scores but remarkable improvements in
automatic and manual evaluations of the generated headlines.
