  Document images are a ubiquitous source of data where the text is organized
in a complex hierarchical structure ranging from fine granularity (e.g.,
words), medium granularity (e.g., regions such as paragraphs or figures), to
coarse granularity (e.g., the whole page). The spatial hierarchical
relationships between content at different levels of granularity are crucial
for document image understanding tasks. Existing methods learn features from
either word-level or region-level but fail to consider both simultaneously.
Word-level models are restricted by the fact that they originate from pure-text
language models, which only encode the word-level context. In contrast,
region-level models attempt to encode regions corresponding to paragraphs or
text blocks into a single embedding, but they perform worse with additional
word-level features. To deal with these issues, we propose MGDoc, a new
multi-modal multi-granular pre-training framework that encodes page-level,
region-level, and word-level information at the same time. MGDoc uses a unified
text-visual encoder to obtain multi-modal features across different
granularities, which makes it possible to project the multi-granular features
into the same hyperspace. To model the region-word correlation, we design a
cross-granular attention mechanism and specific pre-training tasks for our
model to reinforce the model of learning the hierarchy between regions and
words. Experiments demonstrate that our proposed model can learn better
features that perform well across granularities and lead to improvements in
downstream tasks.
