  Recent years have witnessed the rise and success of pre-training techniques
in visually-rich document understanding. However, most existing methods lack
the systematic mining and utilization of layout-centered knowledge, leading to
sub-optimal performances. In this paper, we propose ERNIE-Layout, a novel
document pre-training solution with layout knowledge enhancement in the whole
workflow, to learn better representations that combine the features from text,
layout, and image. Specifically, we first rearrange input sequences in the
serialization stage, and then present a correlative pre-training task, reading
order prediction, to learn the proper reading order of documents. To improve
the layout awareness of the model, we integrate a spatial-aware disentangled
attention into the multi-modal transformer and a replaced regions prediction
task into the pre-training phase. Experimental results show that ERNIE-Layout
achieves superior performance on various downstream tasks, setting new
state-of-the-art on key information extraction, document image classification,
and document question answering datasets. The code and models are publicly
available at
http://github.com/PaddlePaddle/PaddleNLP/tree/develop/model_zoo/ernie-layout.
