  Large, pre-trained transformer models like BERT have achieved
state-of-the-art results on document understanding tasks, but most
implementations can only consider 512 tokens at a time. For many real-world
applications, documents can be much longer, and the segmentation strategies
typically used on longer documents miss out on document structure and
contextual information, hurting their results on downstream tasks. In our work
on legal agreements, we find that visual cues such as layout, style, and
placement of text in a document are strong features that are crucial to
achieving an acceptable level of accuracy on long documents. We measure the
impact of incorporating such visual cues, obtained via computer vision methods,
on the accuracy of document understanding tasks including document
segmentation, entity extraction, and attribute classification. Our method of
segmenting documents based on structural metadata out-performs existing methods
on four long-document understanding tasks as measured on the Contract
Understanding Atticus Dataset.
