  Several multi-modality representation learning approaches such as LXMERT and
ViLBERT have been proposed recently. Such approaches can achieve superior
performance due to the high-level semantic information captured during
large-scale multimodal pretraining. However, as ViLBERT and LXMERT adopt visual
region regression and classification loss, they often suffer from domain gap
and noisy label problems, based on the visual features having been pretrained
on the Visual Genome dataset. To overcome these issues, we propose unbiased
Contrastive Visual-Linguistic Pretraining (CVLP), which constructs a visual
self-supervised loss built upon contrastive learning. We evaluate CVLP on
several down-stream tasks, including VQA, GQA and NLVR2 to validate the
superiority of contrastive learning on multi-modality representation learning.
Our code is available at: https://github.com/ArcherYunDong/CVLP-.
