  This paper investigates the Relation Extraction task in documents by
benchmarking two different neural network models: a multi-modal language model
(LayoutXLM) and a Graph Neural Network: Edge Convolution Network (ECN). For
this benchmark, we use the XFUND dataset, released along with LayoutXLM. While
both models reach similar results, they both exhibit very different
characteristics. This raises the question on how to integrate various
modalities in a neural network: by merging all modalities thanks to additional
pretraining (LayoutXLM), or in a cascaded way (ECN). We conclude by discussing
some methodological issues that must be considered for new datasets and task
definition in the domain of Information Extraction with complex documents.
