  Since their release, Transformers have revolutionized many fields from
Natural Language Understanding to Computer Vision. Document Understanding (DU)
was not left behind with first Transformer based models for DU dating from late
2019. However, the computational complexity of the self-attention operation
limits their capabilities to small sequences. In this paper we explore multiple
strategies to apply Transformer based models to long multi-page documents. We
introduce 2 new multi-modal (text + layout) long-range models for DU. They are
based on efficient implementations of Transformers for long sequences.
Long-range models can process whole documents at once effectively and are less
impaired by the document's length. We compare them to LayoutLM, a classical
Transformer adapted for DU and pre-trained on millions of documents. We further
propose 2D relative attention bias to guide self-attention towards relevant
tokens without harming model efficiency. We observe improvements on multi-page
business documents on Information Retrieval for a small performance cost on
smaller sequences. Relative 2D attention revealed to be effective on dense text
for both normal and long-range models.
