  It is highly desirable yet challenging to generate image captions that can
describe novel objects which are unseen in caption-labeled training data, a
capability that is evaluated in the novel object captioning challenge (nocaps).
In this challenge, no additional image-caption training data, other thanCOCO
Captions, is allowed for model training. Thus, conventional Vision-Language
Pre-training (VLP) methods cannot be applied. This paper presents VIsual
VOcabulary pretraining (VIVO) that performs pre-training in the absence of
caption annotations. By breaking the dependency of paired image-caption
training data in VLP, VIVO can leverage large amounts of paired image-tag data
to learn a visual vocabulary. This is done by pre-training a multi-layer
Transformer model that learns to align image-level tags with their
corresponding image region features. To address the unordered nature of image
tags, VIVO uses a Hungarian matching loss with masked tag prediction to conduct
pre-training. We validate the effectiveness of VIVO by fine-tuning the
pre-trained model for image captioning. In addition, we perform an analysis of
the visual-text alignment inferred by our model. The results show that our
model can not only generate fluent image captions that describe novel objects,
but also identify the locations of these objects. Our single model has achieved
new state-of-the-art results on nocaps and surpassed the human CIDEr score.
