  Multimodal learning from document data has achieved great success lately as
it allows to pre-train semantically meaningful features as a prior into a
learnable downstream task. In this paper, we approach the document
classification problem by learning cross-modal representations through language
and vision cues, considering intra- and inter-modality relationships. Instead
of merging features from different modalities into a joint representation
space, the proposed method exploits high-level interactions and learns relevant
semantic information from effective attention flows within and across
modalities. The proposed learning objective is devised between intra- and
inter-modality alignment tasks, where the similarity distribution per task is
computed by contracting positive sample pairs while simultaneously contrasting
negative ones in the joint representation space}. Extensive experiments on
public document classification datasets demonstrate the effectiveness and the
generality of our model on low-scale and large-scale datasets.
