  Vision-language pre-training has been an emerging and fast-developing
research topic, which transfers multi-modal knowledge from rich-resource
pre-training task to limited-resource downstream tasks. Unlike existing works
that predominantly learn a single generic encoder, we present a pre-trainable
Universal Encoder-DEcoder Network (Uni-EDEN) to facilitate both vision-language
perception (e.g., visual question answering) and generation (e.g., image
captioning). Uni-EDEN is a two-stream Transformer based structure, consisting
of three modules: object and sentence encoders that separately learns the
representations of each modality, and sentence decoder that enables both
multi-modal reasoning and sentence generation via inter-modal interaction.
Considering that the linguistic representations of each image can span
different granularities in this hierarchy including, from simple to
comprehensive, individual label, a phrase, and a natural sentence, we pre-train
Uni-EDEN through multi-granular vision-language proxy tasks: Masked Object
Classification (MOC), Masked Region Phrase Generation (MRPG), Image-Sentence
Matching (ISM), and Masked Sentence Generation (MSG). In this way, Uni-EDEN is
endowed with the power of both multi-modal representation extraction and
language modeling. Extensive experiments demonstrate the compelling
generalizability of Uni-EDEN by fine-tuning it to four vision-language
perception and generation downstream tasks.
