  In this paper we apply self-knowledge distillation to text summarization
which we argue can alleviate problems with maximum-likelihood training on
single reference and noisy datasets. Instead of relying on one-hot annotation
labels, our student summarization model is trained with guidance from a teacher
which generates smoothed labels to help regularize training. Furthermore, to
better model uncertainty during training, we introduce multiple noise signals
for both teacher and student models. We demonstrate experimentally on three
benchmarks that our framework boosts the performance of both pretrained and
non-pretrained summarizers achieving state-of-the-art results.
