  Recent years have witnessed the fast development of large-scale pre-training
frameworks that can extract multi-modal representations in a unified form and
achieve promising performances when transferred to downstream tasks.
Nevertheless, existing approaches mainly focus on pre-training with simple
image-text pairs, while neglecting the semantic connections between concepts
from different modalities. In this paper, we propose a knowledge-based
pre-training framework, dubbed Knowledge-CLIP, which injects semantic
information into the widely used CLIP model. Through introducing
knowledge-based objectives in the pre-training process and utilizing different
types of knowledge graphs as training data, our model can semantically align
the representations in vision and language with higher quality, and enhance the
reasoning ability across scenarios and modalities. Extensive experiments on
various vision-language downstream tasks demonstrate the effectiveness of
Knowledge-CLIP compared with the original CLIP and competitive baselines.
