  Transformer-based pre-training techniques of text and layout have proven
effective in a number of document understanding tasks. Despite this success,
multimodal pre-training models suffer from very high computational and memory
costs. Motivated by human reading strategies, this paper presents
Skim-Attention, a new attention mechanism that takes advantage of the structure
of the document and its layout. Skim-Attention only attends to the
2-dimensional position of the words in a document. Our experiments show that
Skim-Attention obtains a lower perplexity than prior works, while being more
computationally efficient. Skim-Attention can be further combined with
long-range Transformers to efficiently process long documents. We also show how
Skim-Attention can be used off-the-shelf as a mask for any Pre-trained Language
Model, allowing to improve their performance while restricting attention.
Finally, we show the emergence of a document structure representation in
Skim-Attention.
