  Automatically generating textual descriptions for massive unlabeled images on
the web can greatly benefit realistic web applications, e.g. multimodal
retrieval and recommendation. However, existing models suffer from the problem
of generating ``over-generic'' descriptions, such as their tendency to generate
repetitive sentences with common concepts for different images. These generic
descriptions fail to provide sufficient textual semantics for ever-changing web
images. Inspired by the recent success of Vision-Language Pre-training (VLP)
models that learn diverse image-text concept alignment during pretraining, we
explore leveraging their cross-modal pre-trained knowledge to automatically
enrich the textual semantics of image descriptions. With no need for additional
human annotations, we propose a plug-and-play framework, i.e CapEnrich, to
complement the generic image descriptions with more semantic details.
Specifically, we first propose an automatic data-building strategy to get
desired training sentences, based on which we then adopt prompting strategies,
i.e. learnable and template prompts, to incentivize VLP models to generate more
textual details. For learnable templates, we fix the whole VLP model and only
tune the prompt vectors, which leads to two advantages: 1) the pre-training
knowledge of VLP models can be reserved as much as possible to describe diverse
visual concepts; 2) only lightweight trainable parameters are required, so it
is friendly to low data resources. Extensive experiments show that our method
significantly improves the descriptiveness and diversity of generated sentences
for web images. The code is available at https://github.com/yaolinli/CapEnrich.
