  Multimodal fusion breaks through the barriers between diverse modalities and
has already yielded numerous impressive performances. However, in various
specialized fields, it is struggling to obtain sufficient alignment data for
the training process, which seriously limits the use of previously elegant
models. Thus, semi-supervised learning attempts to achieve multimodal alignment
with fewer matched pairs but traditional methods like pseudo-labeling are
difficult to apply in domains with no label information. To address these
problems, we transform semi-supervised multimodal alignment into a manifold
matching problem and propose a new method based on CLIP, named Gentle-CLIP.
Specifically, we design a novel semantic density distribution loss to explore
implicit semantic alignment information from unpaired multimodal data by
constraining the latent representation distribution with fine granularity, thus
eliminating the need for numerous strictly matched pairs. Meanwhile, we
introduce multi-kernel maximum mean discrepancy as well as self-supervised
contrastive loss to pull separate modality distributions closer and enhance the
stability of the representation distribution. In addition, the contrastive loss
used in CLIP is employed on the supervised matched data to prevent negative
optimization. Extensive experiments conducted on a range of tasks in various
fields, including protein, remote sensing, and the general vision-language
field, demonstrate the effectiveness of our proposed Gentle-CLIP.
