  The Hierarchical Attention Network (HAN) has made great strides, but it
suffers a major limitation: at level 1, each sentence is encoded in complete
isolation. In this work, we propose and compare several modifications of HAN in
which the sentence encoder is able to make context-aware attentional decisions
(CAHAN). Furthermore, we propose a bidirectional document encoder that
processes the document forwards and backwards, using the preceding and
following sentences as context. Experiments on three large-scale sentiment and
topic classification datasets show that the bidirectional version of CAHAN
outperforms HAN everywhere, with only a modest increase in computation time.
While results are promising, we expect the superiority of CAHAN to be even more
evident on tasks requiring a deeper understanding of the input documents, such
as abstractive summarization. Code is publicly available.
