  Evaluation of a document summarization system has been a critical factor to
impact the success of the summarization task. Previous approaches, such as
ROUGE, mainly consider the informativeness of the assessed summary and require
human-generated references for each test summary. In this work, we propose to
evaluate the summary qualities without reference summaries by unsupervised
contrastive learning. Specifically, we design a new metric which covers both
linguistic qualities and semantic informativeness based on BERT. To learn the
metric, for each summary, we construct different types of negative samples with
respect to different aspects of the summary qualities, and train our model with
a ranking loss. Experiments on Newsroom and CNN/Daily Mail demonstrate that our
new evaluation method outperforms other metrics even without reference
summaries. Furthermore, we show that our method is general and transferable
across datasets.
