  Text summarization aims to condense long documents and retain key
information. Critical to the success of a summarization model is the faithful
inference of latent representations of words or tokens in the source documents.
Most recent models infer the latent representations with a transformer encoder,
which is purely bottom-up. Also, self-attention-based inference models face the
challenge of quadratic complexity with respect to sequence length. We propose a
principled inference framework to improve summarization models on these two
aspects. Our framework assumes a hierarchical latent structure of a document
where the top-level captures the long range dependency at a coarser time scale
and the bottom token level preserves the details. Critically, this hierarchical
structure enables token representations to be updated in both a bottom-up and
top-down manner. In the bottom-up pass, token representations are inferred with
local self-attention to leverage its efficiency. Top-down correction is then
applied to allow tokens to capture long-range dependency. We demonstrate the
effectiveness of the proposed framework on a diverse set of summarization
datasets, including narrative, conversational, scientific documents and news.
Our model achieves (1) competitive or better performance on short documents
with higher memory and compute efficiency, compared to full attention
transformers, and (2) state-of-the-art performance on a wide range of long
document summarization benchmarks, compared to recent efficient transformers.
We also show that our model can summarize an entire book and achieve
competitive performance using $0.27\%$ parameters (464M vs. 175B) and much less
training data, compared to a recent GPT-3-based model. These results indicate
the general applicability and benefits of the proposed framework.
