  Multimodal integration of text, layout and visual information has achieved
SOTA results in visually rich document understanding (VrDU) tasks, including
relation extraction (RE). However, despite its importance, evaluation of the
relative predictive capacity of these modalities is less prevalent. Here, we
demonstrate the value of shared representations for RE tasks by conducting
experiments in which each data type is iteratively excluded during training. In
addition, text and layout data are evaluated in isolation. While a bimodal text
and layout approach performs best (F1=0.684), we show that text is the most
important single predictor of entity relations. Additionally, layout geometry
is highly predictive and may even be a feasible unimodal approach. Despite
being less effective, we highlight circumstances where visual information can
bolster performance. In total, our results demonstrate the efficacy of training
joint representations for RE.
