  Document intelligence automates the extraction of information from documents
and supports many business applications. Recent self-supervised learning
methods on large-scale unlabeled document datasets have opened up promising
directions towards reducing annotation efforts by training models with
self-supervised objectives. However, most of the existing document pretraining
methods are still language-dominated. We present UDoc, a new unified
pretraining framework for document understanding. UDoc is designed to support
most document understanding tasks, extending the Transformer to take multimodal
embeddings as input. Each input element is composed of words and visual
features from a semantic region of the input document image. An important
feature of UDoc is that it learns a generic representation by making use of
three self-supervised losses, encouraging the representation to model
sentences, learn similarities, and align modalities. Extensive empirical
analysis demonstrates that the pretraining procedure learns better joint
representations and leads to improvements in downstream tasks.
