  Contrastive Language-Image Pre-training (CLIP) excels in integrating semantic
information between images and text through contrastive learning techniques. It
has achieved remarkable performance in various multimodal tasks. However, the
deployment of large CLIP models is hindered in resource-limited environments,
while smaller models frequently fall short of meeting performance benchmarks
necessary for practical applications. In this paper, we propose a novel
approach, coined as ComKD-CLIP: Comprehensive Knowledge Distillation for
Contrastive Language-Image Pre-traning Model, which aims to comprehensively
distill the knowledge from a large teacher CLIP model into a smaller student
model, ensuring comparable performance with significantly reduced parameters.
ComKD-CLIP is composed of two key mechanisms: Image Feature Alignment (IFAlign)
and Educational Attention (EduAttention). IFAlign makes the image features
extracted by the student model closely match those extracted by the teacher
model, enabling the student to learn teacher's knowledge of extracting image
features. EduAttention explores the cross-relationships between text features
extracted by the teacher model and image features extracted by the student
model, enabling the student model to learn how the teacher model integrates
text-image features. In addition, ComKD-CLIP can refine the knowledge distilled
from IFAlign and EduAttention leveraging the results of text-image feature
fusion by the teacher model, ensuring student model accurately absorbs the
knowledge of teacher model. Extensive experiments conducted on 11 datasets have
demonstrated the superiority of the proposed method.
