  Unsupervised extractive document summarization aims to select important
sentences from a document without using labeled summaries during training.
Existing methods are mostly graph-based with sentences as nodes and edge
weights measured by sentence similarities. In this work, we find that
transformer attentions can be used to rank sentences for unsupervised
extractive summarization. Specifically, we first pre-train a hierarchical
transformer model using unlabeled documents only. Then we propose a method to
rank sentences using sentence-level self-attentions and pre-training
objectives. Experiments on CNN/DailyMail and New York Times datasets show our
model achieves state-of-the-art performance on unsupervised summarization. We
also find in experiments that our model is less dependent on sentence
positions. When using a linear combination of our model and a recent
unsupervised model explicitly modeling sentence positions, we obtain even
better results.
