  Multi-modal representation learning by pretraining has become an increasing
interest due to its easy-to-use and potential benefit for various
Visual-and-Language~(V-L) tasks. However its requirement of large volume and
high-quality vision-language pairs highly hinders its values in practice. In
this paper, we proposed a novel label-augmented V-L pretraining model, named
LAMP, to address this problem. Specifically, we leveraged auto-generated labels
of visual objects to enrich vision-language pairs with fine-grained alignment
and correspondingly designed a novel pretraining task. Besides, we also found
such label augmentation in second-stage pretraining would further universally
benefit various downstream tasks. To evaluate LAMP, we compared it with some
state-of-the-art models on four downstream tasks. The quantitative results and
analysis have well proven the value of labels in V-L pretraining and the
effectiveness of LAMP.
