  Sequence modeling has demonstrated state-of-the-art performance on natural
language and document understanding tasks. However, it is challenging to
correctly serialize tokens in form-like documents in practice due to their
variety of layout patterns. We propose FormNet, a structure-aware sequence
model to mitigate the suboptimal serialization of forms. First, we design Rich
Attention that leverages the spatial relationship between tokens in a form for
more precise attention score calculation. Second, we construct Super-Tokens for
each word by embedding representations from their neighboring tokens through
graph convolutions. FormNet therefore explicitly recovers local syntactic
information that may have been lost during serialization. In experiments,
FormNet outperforms existing methods with a more compact model size and less
pre-training data, establishing new state-of-the-art performance on CORD, FUNSD
and Payment benchmarks.
